Reproducible documentation of analyses

Does Computer-Based Feedback Enhance Revision Performance in Writing Non-Fictional Texts? A Meta-Analysis

Author

Affiliation

[blinded]

[blinded]

Display code

We folded the code for better readability.
You can display the code of our analyses

1. by clicking on “Code” > “Show All Code” in the top right corner or
2. by clicking in “Code” in each section of the document individually

Code

library(tidyverse)
library(here)
library(haven)
library(metafor)
library(kableExtra)
library(xfun)
library(skimr)

Import Data Sets

Two data sets were used:

• Data set “dataN33_prepost”: studies with pre-post design
• Data set “dataN14_control”: studies with pre-post-control design

Download the data (embeded in this HTML file):

• Download dataN33_prepost
• Download dataN14_control

Code

# Data set: studies with pre-post design
dataN33_prepost <- read_sav(here("02_data/05_major revision/dataN33_prepost.sav"))

# Data set: studies with pre-post-control design
dataN14_control <- read_sav(here("02_data/05_major revision/dataN14_control.sav"))

Pre-Post comparison

Code

# prepare data set for meta-analysis
data_pp <- data.frame(
  sampleNr = 1:33,
  id = c(1,1,1,2,2,3,3,4,4,4,4,4,4,5,5,6,7,7,8,8,9,10,11,12,12,13,14,14,14,15,15, 16, 17),
  m_pre = dataN33_prepost$M_pre,
  m_post = dataN33_prepost$M_post,
  sd_pre = dataN33_prepost$SD_pre,
  sd_post = dataN33_prepost$SD_post,
  ni = dataN33_prepost$N,
  slab = dataN33_prepost$author)

Meta-analytic effect

As we did not find the pre-post test correlation reported in any of the studies, we ran a sensitivity analysis.

One of the studies with control group design provided open data (Burkhart et al., 2020) so that we were able to calculate its pre-post-test correlations (r = .72), which we used as an anchor for the sensitivity analysis.

We included a broad range of 26 (conservative) plausible values in the sensitivity analyses with correlations from r= .50 - .75, using steps of .01.

This will produce results from 26 meta-analyses. By the variation of results, we are able to estimate the robustness of our analyses.

Code

# Establish empty data frame to be filled with results
sensitivity <- data.frame(ri_t = as.numeric(),   # assumed pre-post correlation
                          beta = as.numeric(),   # meta-analytic ES
                          pvalue = as.numeric(), # p value of ES
                          se = as.numeric(),     # SE of meta-analytic ES
                          sigma2 = as.numeric(),   # Tau squared
                          yi.f01 = as.numeric(), # ES of individual studies
                          yi.f02 = as.numeric(),
                          yi.f03 = as.numeric(),
                          yi.f04 = as.numeric(),
                          yi.f05 = as.numeric(),
                          yi.f06 = as.numeric(),
                          yi.f07 = as.numeric(),
                          yi.f08 = as.numeric(),
                          yi.f09 = as.numeric(),
                          yi.f10 = as.numeric(),
                          yi.f11 = as.numeric(),
                          yi.f12 = as.numeric(),
                          yi.f13 = as.numeric(),
                          yi.f14 = as.numeric(),
                          yi.f15 = as.numeric(),
                          yi.f16 = as.numeric(),
                          yi.f17 = as.numeric(),
                          yi.f18 = as.numeric(),
                          yi.f19 = as.numeric(),
                          yi.f20 = as.numeric(),
                          yi.f21 = as.numeric(),
                          yi.f22 = as.numeric(),
                          yi.f23 = as.numeric(),
                          yi.f24 = as.numeric(),
                          yi.f25 = as.numeric(),
                          yi.f26 = as.numeric(),
                          yi.f27 = as.numeric(),
                          yi.f28 = as.numeric(),
                          yi.f29 = as.numeric(),
                          yi.f30 = as.numeric(),
                          yi.f31 = as.numeric(),
                          yi.f32 = as.numeric(),
                          yi.f33 = as.numeric(),
                          sei.f01 = as.numeric(),  # SEs of ES of individual studies
                          sei.f02 = as.numeric(),
                          sei.f03 = as.numeric(),
                          sei.f04 = as.numeric(),
                          sei.f05 = as.numeric(),
                          sei.f06 = as.numeric(),
                          sei.f07 = as.numeric(),
                          sei.f08 = as.numeric(),
                          sei.f09 = as.numeric(),
                          sei.f10 = as.numeric(),
                          sei.f11 = as.numeric(),
                          sei.f12 = as.numeric(),
                          sei.f13 = as.numeric(),
                          sei.f14 = as.numeric(),
                          sei.f15 = as.numeric(),
                          sei.f16 = as.numeric(),
                          sei.f17 = as.numeric(),
                          sei.f18 = as.numeric(),
                          sei.f19 = as.numeric(),
                          sei.f20 = as.numeric(),
                          sei.f21 = as.numeric(),
                          sei.f22 = as.numeric(),
                          sei.f23 = as.numeric(),
                          sei.f24 = as.numeric(),
                          sei.f25 = as.numeric(),
                          sei.f26 = as.numeric(),
                          sei.f27 = as.numeric(),
                          sei.f28 = as.numeric(),
                          sei.f29 = as.numeric(),
                          sei.f30 = as.numeric(),
                          sei.f31 = as.numeric(),
                          sei.f32 = as.numeric(),
                          sei.f33 = as.numeric()
)

# starting loop over 26 possible pre-post-correlations
for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  
  # compute the meta-analysis
  rma_overall_clustered <- rma.mv(yi, vi, 
                                  data=data_pp_es, 
                                  random = ~ 1 | id # take clustered data into account
                                  )
  
  # save estimates for sensitivity analysis
  sensitivity <- sensitivity %>%
    add_row(ri_t = ri_t, 
            pvalue = rma_overall_clustered$pval,
            beta = rma_overall_clustered$beta[,1],
            se = rma_overall_clustered$se[1],
            sigma2 = rma_overall_clustered$sigma2[1],
            yi.f01 = rma_overall_clustered$yi.f[1],
            yi.f02 = rma_overall_clustered$yi.f[2],
            yi.f03 = rma_overall_clustered$yi.f[3],
            yi.f04 = rma_overall_clustered$yi.f[4],
            yi.f05 = rma_overall_clustered$yi.f[5],
            yi.f06 = rma_overall_clustered$yi.f[6],
            yi.f07 = rma_overall_clustered$yi.f[7],
            yi.f08 = rma_overall_clustered$yi.f[8],
            yi.f09 = rma_overall_clustered$yi.f[9],
            yi.f10 = rma_overall_clustered$yi.f[10],
            yi.f11 = rma_overall_clustered$yi.f[11],
            yi.f12 = rma_overall_clustered$yi.f[12],
            yi.f13 = rma_overall_clustered$yi.f[13],
            yi.f14 = rma_overall_clustered$yi.f[14],
            yi.f15 = rma_overall_clustered$yi.f[15],
            yi.f16 = rma_overall_clustered$yi.f[16],
            yi.f17 = rma_overall_clustered$yi.f[17],
            yi.f18 = rma_overall_clustered$yi.f[18],
            yi.f19 = rma_overall_clustered$yi.f[19],
            yi.f20 = rma_overall_clustered$yi.f[20],
            yi.f21 = rma_overall_clustered$yi.f[21],
            yi.f22 = rma_overall_clustered$yi.f[22],
            yi.f23 = rma_overall_clustered$yi.f[23],
            yi.f24 = rma_overall_clustered$yi.f[24],
            yi.f25 = rma_overall_clustered$yi.f[25],
            yi.f26 = rma_overall_clustered$yi.f[26],
            yi.f27 = rma_overall_clustered$yi.f[27],
            yi.f28 = rma_overall_clustered$yi.f[28],
            yi.f29 = rma_overall_clustered$yi.f[29],
            yi.f30 = rma_overall_clustered$yi.f[30],
            yi.f31 = rma_overall_clustered$yi.f[31],
            yi.f32 = rma_overall_clustered$yi.f[32],
            yi.f33 = rma_overall_clustered$yi.f[33],
            sei.f01 = sqrt(data_pp_es$vi[1]),
            sei.f02 = sqrt(data_pp_es$vi[2]),
            sei.f03 = sqrt(data_pp_es$vi[3]),
            sei.f04 = sqrt(data_pp_es$vi[4]),
            sei.f05 = sqrt(data_pp_es$vi[5]),
            sei.f06 = sqrt(data_pp_es$vi[6]),
            sei.f07 = sqrt(data_pp_es$vi[7]),
            sei.f08 = sqrt(data_pp_es$vi[8]),
            sei.f09 = sqrt(data_pp_es$vi[9]),
            sei.f10 = sqrt(data_pp_es$vi[10]),
            sei.f11 = sqrt(data_pp_es$vi[11]),
            sei.f12 = sqrt(data_pp_es$vi[12]),
            sei.f13 = sqrt(data_pp_es$vi[13]),
            sei.f14 = sqrt(data_pp_es$vi[14]),
            sei.f15 = sqrt(data_pp_es$vi[15]),
            sei.f16 = sqrt(data_pp_es$vi[16]),
            sei.f17 = sqrt(data_pp_es$vi[17]),
            sei.f18 = sqrt(data_pp_es$vi[18]),
            sei.f19 = sqrt(data_pp_es$vi[19]),
            sei.f20 = sqrt(data_pp_es$vi[20]),
            sei.f21 = sqrt(data_pp_es$vi[21]),
            sei.f22 = sqrt(data_pp_es$vi[22]),
            sei.f23 = sqrt(data_pp_es$vi[23]),
            sei.f24 = sqrt(data_pp_es$vi[24]),
            sei.f25 = sqrt(data_pp_es$vi[25]),
            sei.f26 = sqrt(data_pp_es$vi[26]),
            sei.f27 = sqrt(data_pp_es$vi[27]),
            sei.f28 = sqrt(data_pp_es$vi[28]),
            sei.f29 = sqrt(data_pp_es$vi[29]),
            sei.f30 = sqrt(data_pp_es$vi[30]),
            sei.f31 = sqrt(data_pp_es$vi[31]),
            sei.f32 = sqrt(data_pp_es$vi[32]),
            sei.f33 = sqrt(data_pp_es$vi[33])
    )
}

Overview of meta-analytic ES

Results from all 26 meta-analyses:

• ri_t: assumed pre-post-correlation
• beta: meta-analytic ES
• se: SE of meta-analytic ES
• pvalue: p value of meta-analytic ES

Code

sensitivity %>%
  dplyr::select(ri_t, beta, se, pvalue, sigma2) %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	pvalue	sigma2
0.50	0.4609693	0.1568736	0.0032983	0.3977544
0.51	0.4609604	0.1568797	0.0033001	0.3980426
0.52	0.4609474	0.1568848	0.0033020	0.3983259
0.53	0.4609302	0.1568888	0.0033040	0.3986040
0.54	0.4609085	0.1568918	0.0033061	0.3988767
0.55	0.4608822	0.1568937	0.0033082	0.3991438
0.56	0.4608511	0.1568944	0.0033105	0.3994050
0.57	0.4608150	0.1568940	0.0033129	0.3996600
0.58	0.4607737	0.1568922	0.0033153	0.3999087
0.59	0.4607270	0.1568891	0.0033179	0.4001507
0.60	0.4606747	0.1568847	0.0033206	0.4003856
0.61	0.4606165	0.1568788	0.0033234	0.4006133
0.62	0.4605521	0.1568714	0.0033263	0.4008332
0.63	0.4604813	0.1568625	0.0033293	0.4010451
0.64	0.4604038	0.1568518	0.0033325	0.4012485
0.65	0.4603192	0.1568395	0.0033358	0.4014431
0.66	0.4602273	0.1568253	0.0033392	0.4016284
0.67	0.4601277	0.1568092	0.0033428	0.4018038
0.68	0.4600200	0.1567911	0.0033466	0.4019689
0.69	0.4599038	0.1567709	0.0033505	0.4021232
0.70	0.4597787	0.1567485	0.0033546	0.4022660
0.71	0.4596442	0.1567237	0.0033589	0.4023966
0.72	0.4594999	0.1566964	0.0033633	0.4025145
0.73	0.4593452	0.1566664	0.0033679	0.4026188
0.74	0.4591795	0.1566337	0.0033728	0.4027088
0.75	0.4590023	0.1565980	0.0033778	0.4027835

We summarize these results:

• ES_mean: meta-analytic ES (mean over the 26 meta-analyses)
• ES_min: meta-analytic ES (min of the 26 meta-analyses)
• ES_max: meta-analytic ES (max of the 26 meta-analyses)
• CI_lower_mean: lower CI of meta-analytic ES (mean over the 26 meta-analyses)
• CI_lower_min: lower CI of meta-analytic ES (min of the 26 meta-analyses)
• CI_lower_max: lower CI of meta-analytic ES (max of the 26 meta-analyses)
• CI_upper_mean: upper CI of meta-analytic ES (mean over the 26 meta-analyses)
• CI_upper_min: upper CI of meta-analytic ES (min of the 26 meta-analyses)
• CI_upper_max: upper CI of meta-analytic ES (max of the 26 meta-analyses)
• pvalue_mean: p value of meta-analytic ES (mean over the 26 meta-analyses)
• pvalue_min: p value of meta-analytic ES (min of the 26 meta-analyses)
• pvalue_max: p value of meta-analytic ES (max of the 26 meta-analyses)

Code

# compute mean, min and max of ES, CI and pvalue from meta-analysis
sensitivity %>%
  dplyr::summarise(ES_mean = mean(beta),
                   ES_min  = min(beta),
                   ES_max  = max(beta),
                   CI_lower_mean        = mean(beta-(1.96*se)),
                   CI_lower_min = min(beta-(1.96*se)),
                   CI_lower_max = max(beta-(1.96*se)),
                   CI_upper_mean        = mean(beta+(1.96*se)),
                   CI_upper_min = min(beta+(1.96*se)),
                   CI_upper_max = max(beta+(1.96*se)),
                   pvalue_mean        = mean(pvalue),
                   pvalue_min = min(pvalue),
                   pvalue_max = max(pvalue),
                   sigma2_mean = mean(sigma2)) %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover")) %>%
  column_spec(1, background = "#C74A48") %>%
  column_spec(2, background = "#B54341") %>%
  column_spec(3, background = "#913634") %>%
  column_spec(4, background = "#27A357") %>%
  column_spec(5, background = "#208748") %>%
  column_spec(6, background = "#186636") %>%
  column_spec(7, background = "#B244B8") %>%
  column_spec(8, background = "#8A358F") %>%
  column_spec(9, background = "#6A296E") %>%
  column_spec(10, background = "#9E8A47") %>%
  column_spec(11, background = "#8F7C40") %>%
  column_spec(12, background = "#6E6031")

ES_mean	ES_min	ES_max	CI_lower_mean	CI_lower_min	CI_lower_max	CI_upper_mean	CI_upper_min	CI_upper_max	pvalue_mean	pvalue_min	pvalue_max	sigma2_mean
0.4603285	0.4590023	0.4609693	0.1529602	0.1520702	0.1534971	0.7676968	0.7659345	0.7684446	0.0033314	0.0032983	0.0033778	0.400702

We summarize the same aspects for all 33 studies:
Note that the ES does not vary due to different pre-post-correlation, but SE does.

Code

# compute mean, min and max of ES and CI from each study
sensitivity %>%
  dplyr::select(-c(beta, pvalue, se, sigma2)) %>%
  pivot_longer(c(2:67),                       # reshape data from
               names_to = "variable",         # sensitivity analysis
               values_to = "values") %>%
  mutate(sampleNr = as.numeric(str_sub(variable, -2, -1)),
         variable = str_sub(variable, 1, -5)) %>%
  pivot_wider(id_cols = c(sampleNr, ri_t), 
              names_from = "variable", 
              values_from = "values") %>%
  group_by(sampleNr) %>%
  dplyr::summarise(ES = mean(yi),
                   CI_lower_mean = mean(yi-(1.96*sei)),
                   CI_lower_min  = min(yi-(1.96*sei)),
                   CI_lower_max  = max(yi-(1.96*sei)),
                   CI_upper_mean = mean(yi+(1.96*sei)),
                   CI_upper_min  = min(yi+(1.96*sei)),
                   CI_upper_max  = max(yi+(1.96*sei)))  %>%
  right_join(data_pp[c("sampleNr", "slab")],., by = "sampleNr") %>%
  dplyr::select(-sampleNr) %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover")) %>%
  column_spec(2, background = "#C74A48") %>%
  column_spec(3, background = "#27A357") %>%
  column_spec(4, background = "#208748") %>%
  column_spec(5, background = "#186636") %>%
  column_spec(6, background = "#B244B8") %>%
  column_spec(7, background = "#8A358F") %>%
  column_spec(8, background = "#6A296E")

slab	ES	CI_lower_mean	CI_lower_min	CI_lower_max	CI_upper_mean	CI_upper_min	CI_upper_max
Burkhart et al., 2020	1.4692438	0.9394991	0.9039832	0.9766106	1.9989885	1.9618770	2.0345044
Burkhart et al., 2020	0.7175647	0.3085082	0.2592778	0.3618768	1.1266212	1.0732525	1.1758516
Burkhart et al., 2020	0.6060285	0.2358310	0.1878340	0.2882082	0.9762260	0.9238488	1.0242230
Frost, 2008	-0.5270188	-1.0024157	-1.0670867	-0.9315185	-0.0516220	-0.1225191	0.0130491
Frost, 2008	0.3241156	-0.0403714	-0.0951686	0.0203292	0.6886025	0.6279020	0.7433997
Kellogg et al., 2010	2.6877490	1.7725416	1.7465410	1.7990238	3.6029564	3.5764742	3.6289570
Kellogg et al., 2010	2.9482961	1.9590182	1.9349339	1.9834837	3.9375740	3.9131084	3.9616582
Lachner et al., 2017a	0.1305370	-0.0428142	-0.0703551	-0.0121139	0.3038881	0.2731878	0.3314290
Lachner et al., 2017a	0.1320125	-0.0615734	-0.0923213	-0.0272991	0.3255983	0.2913241	0.3563462
Lachner et al., 2017a	0.1904274	-0.0031278	-0.0335017	0.0306819	0.3839826	0.3501730	0.4143566
Lachner et al., 2017a	0.0688945	-0.1037383	-0.1313844	-0.0728916	0.2415273	0.2106806	0.2691734
Lachner et al., 2017a	0.0900085	-0.1029692	-0.1338062	-0.0685713	0.2829863	0.2485883	0.3138232
Lachner et al., 2017a	0.1584356	-0.0344010	-0.0648790	-0.0004473	0.3512723	0.3173186	0.3817503
Lachner et al., 2017b (Study 2)	1.4283419	0.8598146	0.8204439	0.9010153	1.9968692	1.9556685	2.0362399
Lachner et al., 2017b (Study 3)	0.7019564	0.1608947	0.0950690	0.2323198	1.2430181	1.1715929	1.3088438
Lachner & Neuburg, 2019	1.0366435	0.6379179	0.6001713	0.6781094	1.4353692	1.3951777	1.4731158
McCarthy et al., 2019	0.2243693	0.0026534	-0.0318304	0.0409983	0.4460852	0.4077403	0.4805690
McCarthy et al., 2019	0.2002616	-0.0225713	-0.0574544	0.0162464	0.4230945	0.3842768	0.4579775
Palermo, 2017	0.4423406	0.3333490	0.3178205	0.3504521	0.5513322	0.5342292	0.5668608
Palermo, 2017	0.9712657	0.8436314	0.8309127	0.8572206	1.0988999	1.0853107	1.1116186
Roscoe et al., 2013	0.1394781	-0.1564306	-0.2033695	-0.1041167	0.4353869	0.3830729	0.4823257
Wang et al., 2020	0.1457761	0.0035404	-0.0189958	0.0286539	0.2880118	0.2628983	0.3105480
Zhu et al., 2017	0.3516930	0.2036157	0.1816037	0.2279688	0.4997703	0.4754171	0.5217822
Wilson et al., 2023	0.7187023	0.4261396	0.3909572	0.4642767	1.0112650	0.9731279	1.0464473
Wilson et al., 2023	0.7724223	0.4740418	0.4395019	0.5113638	1.0708027	1.0334807	1.1053426
Correnti et al., 2022	0.4140777	0.3046662	0.2888570	0.3221039	0.5234892	0.5060515	0.5392984
Wilson et al., 2022	0.2918232	0.1597651	0.1396677	0.1820579	0.4238813	0.4015885	0.4439788
Wilson et al., 2022	-0.0599990	-0.1833126	-0.2030753	-0.1612600	0.0633146	0.0412621	0.0830773
Wilson et al., 2022	0.0384150	-0.0784551	-0.0972110	-0.0575224	0.1552851	0.1343524	0.1740410
McCarthy et al., 2022	0.1974450	-0.0234412	-0.0580441	0.0150680	0.4183312	0.3798220	0.4529341
McCarthy et al., 2022	0.1570233	-0.0646667	-0.0997150	-0.0256202	0.3787133	0.3396668	0.4137616
Butterfuss et al., 2022	0.2954736	0.0520840	0.0150924	0.0931104	0.5388631	0.4978368	0.5758548
Niloy et al., 2023	-0.6649928	-0.7761696	-0.7900395	-0.7610866	-0.5538160	-0.5688990	-0.5399460

Forest Plot

In order to be able to display a forest plot, we calculated a meta-analysis with the mean assumed pre-post-correlation (r=.625).

Code

## FOREST PLOT with mean correlation
###compute effect sizes for writing quality
# effect sizes from within-subjects data
data_pp_es <- escalc(measure = "SMCR",
                     m1i=data_pp$m_post, m2i=data_pp$m_pre,
                     sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                     ni=data_pp$ni, ri=rep(.625, 33),
                     slab = data_pp$slab)
# append the id variable
data_pp_es$id <- data_pp$id

REM <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id)

###FOREST PLOT
forest(REM)

Bias estimation

Funnel plot

Code

##Funnel plot
funnel(REM, legend = T)

Trim and fill

Code

##Trim & fill
rma_trimfill <- rma(yi, vi, data=data_pp_es) # compute random-effects model without
                                             # clustering, as trimfill() can't handle
                                             # these objects
taf_overall <- trimfill(rma_trimfill)
taf_overall 

Estimated number of missing studies on the left side: 0 (SE = 3.1132)

Random-Effects Model (k = 33; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0.3068 (SE = 0.0833)
tau (square root of estimated tau^2 value):      0.5539
I^2 (total heterogeneity / total variability):   97.05%
H^2 (total variability / sampling variability):  33.86

Test for Heterogeneity:
Q(df = 32) = 602.8700, p-val < .0001

Model Results:

estimate      se    zval    pval   ci.lb   ci.ub      
  0.4364  0.1008  4.3296  <.0001  0.2389  0.6340  *** 

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Code

funnel(taf_overall, legend = T)

Egger’s test

Code

##Egger's test
regtest(rma_trimfill)

Regression Test for Funnel Plot Asymmetry

Model:     mixed-effects meta-regression model
Predictor: standard error

Test for Funnel Plot Asymmetry: z = 6.2669, p < .0001
Limit Estimate (as sei -> 0):   b = -0.2622 (CI: -0.5056, -0.0188)

Heterogeneity

Calculating \(I^2\)

Code

###HETEROGENEITY

# Establish empty data frame to be filled with results
heterogeneity_sen <- data.frame(ri_t = as.numeric(),   # assumed pre-post correlation
                                I2 = as.numeric())     # I²

# starting loop over 26 possible pre-post-correlations
for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  
  # compute the meta-analysis
  REM <- rma.mv(yi, vi, 
                data=data_pp_es, 
                random = ~ 1 | id # take clustered data into account
                )
  
  # Formula
  W <- diag(1/REM$vi)
  X <- model.matrix(REM)
  P <- W-W%*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W
  I2 <- 100*REM$sigma2/(sum(REM$sigma2)+(REM$k - REM$p)/sum(diag(P)))
  
  # save estimates for sensitivity analysis
  heterogeneity_sen <- heterogeneity_sen %>%
    add_row(ri_t = ri_t, 
            I2 = I2)
}

heterogeneity_sen %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	I2
0.50	97.03884
0.51	97.09374
0.52	97.14861
0.53	97.20346
0.54	97.25829
0.55	97.31310
0.56	97.36789
0.57	97.42267
0.58	97.47743
0.59	97.53219
0.60	97.58694
0.61	97.64169
0.62	97.69644
0.63	97.75119
0.64	97.80595
0.65	97.86072
0.66	97.91550
0.67	97.97031
0.68	98.02514
0.69	98.08000
0.70	98.13489
0.71	98.18983
0.72	98.24482
0.73	98.29987
0.74	98.35499
0.75	98.41018

Code

skim(heterogeneity_sen)

Data summary

Name	heterogeneity_sen
Number of rows	26
Number of columns	2
_______________________
Column type frequency:
numeric	2
________________________
Group variables	None

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
ri_t	0	1	0.625	0.076	0.500	0.562	0.625	0.688	0.75	▇▇▇▇▇
I2	0	1	97.724	0.419	97.039	97.382	97.724	98.066	98.41	▇▇▇▇▇

Moderators

We will report results from all 26 meta-analyses for the moderators.

For each moderator we provide

• a table with the main parameters from each meta-analysis
  • ri_t: assumed pre-post-correlation (sensitivity analysis)
  • beta: ES of moderator
  • ci.lb: lower bound CI of ES of moderator
  • ci.ub: upper bound CI of ES of moderator
  • pvalue: p value of ES of moderator
• a summary table of these parameters

Representation

Graphical representation

Code

#graphical representation
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$rep_g <- as.factor(dataN33_prepost$rep_graphical)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = rep_g)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.0475445	0.0900853	-0.1290194	0.2241085	0.5976573	0.3899507
0.51	0.0467906	0.0892401	-0.1281167	0.2216979	0.6000539	0.3903602
0.52	0.0460287	0.0883852	-0.1272032	0.2192606	0.6025246	0.3907663
0.53	0.0452585	0.0875206	-0.1262787	0.2167957	0.6050736	0.3911689
0.54	0.0444797	0.0866458	-0.1253430	0.2143024	0.6077058	0.3915679
0.55	0.0436920	0.0857606	-0.1243956	0.2117797	0.6104262	0.3919630
0.56	0.0428951	0.0848646	-0.1234364	0.2092266	0.6132404	0.3923540
0.57	0.0420885	0.0839574	-0.1224649	0.2066419	0.6161542	0.3927409
0.58	0.0412719	0.0830386	-0.1214808	0.2040247	0.6191743	0.3931233
0.59	0.0404449	0.0821079	-0.1204837	0.2013735	0.6223076	0.3935011
0.60	0.0396071	0.0811649	-0.1194731	0.1986873	0.6255618	0.3938740
0.61	0.0387579	0.0802089	-0.1184488	0.1959645	0.6289455	0.3942417
0.62	0.0378969	0.0792397	-0.1174101	0.1932039	0.6324676	0.3946041
0.63	0.0370236	0.0782566	-0.1163566	0.1904038	0.6361383	0.3949607
0.64	0.0361374	0.0772591	-0.1152877	0.1875625	0.6399687	0.3953114
0.65	0.0352377	0.0762467	-0.1142030	0.1846784	0.6439708	0.3956559
0.66	0.0343239	0.0752185	-0.1131017	0.1817496	0.6481580	0.3959936
0.67	0.0333953	0.0741741	-0.1119833	0.1787739	0.6525453	0.3963244
0.68	0.0324513	0.0731126	-0.1108469	0.1757494	0.6571489	0.3966477
0.69	0.0314909	0.0720333	-0.1096918	0.1726736	0.6619872	0.3969632
0.70	0.0305134	0.0709353	-0.1085172	0.1695440	0.6670805	0.3972705
0.71	0.0295179	0.0698177	-0.1073222	0.1663580	0.6724515	0.3975689
0.72	0.0285035	0.0686794	-0.1061056	0.1631125	0.6781257	0.3978581
0.73	0.0274690	0.0675193	-0.1048664	0.1598044	0.6841318	0.3981373
0.74	0.0264134	0.0663363	-0.1036034	0.1564302	0.6905022	0.3984061
0.75	0.0253354	0.0651290	-0.1023152	0.1529860	0.6972737	0.3986636

Code

summary(sensitivity_moderator) 

      ri_t             beta               se              ci.lb        
 Min.   :0.5000   Min.   :0.02534   Min.   :0.06513   Min.   :-0.1290  
 1st Qu.:0.5625   1st Qu.:0.03173   1st Qu.:0.07230   1st Qu.:-0.1232  
 Median :0.6250   Median :0.03746   Median :0.07875   Median :-0.1169  
 Mean   :0.6250   Mean   :0.03710   Mean   :0.07834   Mean   :-0.1165  
 3rd Qu.:0.6875   3rd Qu.:0.04269   3rd Qu.:0.08464   3rd Qu.:-0.1100  
 Max.   :0.7500   Max.   :0.04754   Max.   :0.09009   Max.   :-0.1023  
     ci.ub            pvalue           sigma2      
 Min.   :0.1530   Min.   :0.5977   Min.   :0.3900  
 1st Qu.:0.1734   1st Qu.:0.6140   1st Qu.:0.3925  
 Median :0.1918   Median :0.6343   Median :0.3948  
 Mean   :0.1906   Mean   :0.6389   Mean   :0.3946  
 3rd Qu.:0.2086   3rd Qu.:0.6608   3rd Qu.:0.3969  
 Max.   :0.2241   Max.   :0.6973   Max.   :0.3987  

Numerical representation

Code

#numeric representation
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$rep_n <- as.factor(dataN33_prepost$rep_numeric)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = rep_n)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.0349218	0.0888484	-0.2090615	0.1392179	0.6942832	0.4016268
0.51	-0.0350664	0.0880240	-0.2075903	0.1374575	0.6903552	0.4018744
0.52	-0.0352110	0.0871894	-0.2060992	0.1356772	0.6863269	0.4021171
0.53	-0.0353556	0.0863444	-0.2045875	0.1338763	0.6821934	0.4023546
0.54	-0.0355002	0.0854885	-0.2030546	0.1320542	0.6779495	0.4025868
0.55	-0.0356448	0.0846215	-0.2014999	0.1302103	0.6735897	0.4028134
0.56	-0.0357895	0.0837430	-0.1999227	0.1283438	0.6691080	0.4030342
0.57	-0.0359341	0.0828526	-0.1983223	0.1264540	0.6644979	0.4032489
0.58	-0.0360789	0.0819500	-0.1966979	0.1245402	0.6597526	0.4034572
0.59	-0.0362237	0.0810347	-0.1950487	0.1226014	0.6548647	0.4036590
0.60	-0.0363685	0.0801062	-0.1933739	0.1206368	0.6498262	0.4038538
0.61	-0.0365135	0.0791642	-0.1916725	0.1186455	0.6446285	0.4040413
0.62	-0.0366585	0.0782081	-0.1899435	0.1166265	0.6392623	0.4042213
0.63	-0.0368036	0.0772373	-0.1881860	0.1145787	0.6337176	0.4043934
0.64	-0.0369489	0.0762514	-0.1863988	0.1125010	0.6279833	0.4045571
0.65	-0.0370943	0.0752496	-0.1845808	0.1103922	0.6220477	0.4047121
0.66	-0.0372398	0.0742314	-0.1827307	0.1082510	0.6158979	0.4048580
0.67	-0.0373855	0.0731960	-0.1808471	0.1060760	0.6095196	0.4049942
0.68	-0.0375314	0.0721427	-0.1789285	0.1038657	0.6028976	0.4051202
0.69	-0.0376774	0.0710707	-0.1769735	0.1016187	0.5960149	0.4052356
0.70	-0.0378237	0.0699792	-0.1749804	0.0993330	0.5888529	0.4053397
0.71	-0.0379701	0.0688671	-0.1729471	0.0970069	0.5813913	0.4054320
0.72	-0.0381168	0.0677335	-0.1708719	0.0946384	0.5736075	0.4055117
0.73	-0.0382636	0.0665772	-0.1687525	0.0922252	0.5654766	0.4055781
0.74	-0.0384107	0.0653970	-0.1665865	0.0897651	0.5569708	0.4056305
0.75	-0.0385580	0.0641916	-0.1643714	0.0872553	0.5480592	0.4056679

Code

summary(sensitivity_moderator) 

      ri_t             beta                se              ci.lb        
 Min.   :0.5000   Min.   :-0.03856   Min.   :0.06419   Min.   :-0.2091  
 1st Qu.:0.5625   1st Qu.:-0.03764   1st Qu.:0.07134   1st Qu.:-0.1995  
 Median :0.6250   Median :-0.03673   Median :0.07772   Median :-0.1891  
 Mean   :0.6250   Mean   :-0.03673   Mean   :0.07730   Mean   :-0.1882  
 3rd Qu.:0.6875   3rd Qu.:-0.03583   3rd Qu.:0.08352   3rd Qu.:-0.1775  
 Max.   :0.7500   Max.   :-0.03492   Max.   :0.08885   Max.   :-0.1644  
     ci.ub             pvalue           sigma2      
 Min.   :0.08726   Min.   :0.5481   Min.   :0.4016  
 1st Qu.:0.10218   1st Qu.:0.5977   1st Qu.:0.4031  
 Median :0.11560   Median :0.6365   Median :0.4043  
 Mean   :0.11476   Mean   :0.6311   Mean   :0.4041  
 3rd Qu.:0.12787   3rd Qu.:0.6680   3rd Qu.:0.4052  
 Max.   :0.13922   Max.   :0.6943   Max.   :0.4057  

Highlighting representation

Code

#highlighting representation
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$rep_h <- as.factor(dataN33_prepost$rep_highlighting)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = rep_h)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.0291142	0.1151879	-0.1966498	0.2548783	0.8004581	0.3963487
0.51	0.0283114	0.1141681	-0.1954540	0.2520769	0.8041497	0.3966994
0.52	0.0274993	0.1131361	-0.1942433	0.2492420	0.8079556	0.3970460
0.53	0.0266776	0.1120914	-0.1930174	0.2463727	0.8118819	0.3973886
0.54	0.0258460	0.1110335	-0.1917757	0.2434677	0.8159351	0.3977267
0.55	0.0250041	0.1099622	-0.1905179	0.2405260	0.8201223	0.3980603
0.56	0.0241515	0.1088769	-0.1892434	0.2375464	0.8244508	0.3983892
0.57	0.0232880	0.1077773	-0.1879516	0.2345277	0.8289289	0.3987131
0.58	0.0224132	0.1066628	-0.1866421	0.2314685	0.8335651	0.3990318
0.59	0.0215267	0.1055330	-0.1853142	0.2283675	0.8383688	0.3993450
0.60	0.0206280	0.1043872	-0.1839672	0.2252232	0.8433500	0.3996526
0.61	0.0197167	0.1032250	-0.1826005	0.2220340	0.8485197	0.3999541
0.62	0.0187925	0.1020457	-0.1812134	0.2187984	0.8538897	0.4002493
0.63	0.0178548	0.1008487	-0.1798049	0.2155146	0.8594726	0.4005379
0.64	0.0169032	0.0996332	-0.1783744	0.2121807	0.8652823	0.4008195
0.65	0.0159371	0.0983986	-0.1769207	0.2087948	0.8713339	0.4010937
0.66	0.0149560	0.0971441	-0.1754429	0.2053549	0.8776438	0.4013602
0.67	0.0139594	0.0958687	-0.1739398	0.2018586	0.8842298	0.4016186
0.68	0.0129466	0.0945717	-0.1724104	0.1983037	0.8911116	0.4018683
0.69	0.0119172	0.0932519	-0.1708532	0.1946876	0.8983106	0.4021089
0.70	0.0108704	0.0919084	-0.1692667	0.1910075	0.9058503	0.4023398
0.71	0.0098056	0.0905400	-0.1676495	0.1872607	0.9137566	0.4025605
0.72	0.0087221	0.0891454	-0.1659996	0.1834439	0.9220581	0.4027703
0.73	0.0076192	0.0877233	-0.1643153	0.1795537	0.9307866	0.4029685
0.74	0.0064962	0.0862722	-0.1625942	0.1755866	0.9399771	0.4031545
0.75	0.0053522	0.0847905	-0.1608340	0.1715385	0.9496686	0.4033273

Code

summary(sensitivity_moderator)

      ri_t             beta                se              ci.lb        
 Min.   :0.5000   Min.   :0.005352   Min.   :0.08479   Min.   :-0.1966  
 1st Qu.:0.5625   1st Qu.:0.012175   1st Qu.:0.09358   1st Qu.:-0.1889  
 Median :0.6250   Median :0.018324   Median :0.10145   Median :-0.1805  
 Mean   :0.6250   Mean   :0.017935   Mean   :0.10093   Mean   :-0.1799  
 3rd Qu.:0.6875   3rd Qu.:0.023936   3rd Qu.:0.10860   3rd Qu.:-0.1712  
 Max.   :0.7500   Max.   :0.029114   Max.   :0.11519   Max.   :-0.1608  
     ci.ub            pvalue           sigma2      
 Min.   :0.1715   Min.   :0.8005   Min.   :0.3963  
 1st Qu.:0.1956   1st Qu.:0.8256   1st Qu.:0.3985  
 Median :0.2172   Median :0.8567   Median :0.4004  
 Mean   :0.2158   Mean   :0.8631   Mean   :0.4002  
 3rd Qu.:0.2368   3rd Qu.:0.8965   3rd Qu.:0.4020  
 Max.   :0.2549   Max.   :0.9497   Max.   :0.4033  

Text-based representation

Code

#text-based representation
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$rep_t <- as.factor(dataN33_prepost$rep_text_based)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = rep_t)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.0504044	0.0931938	-0.1322521	0.2330608	0.5886071	0.4074751
0.51	0.0507746	0.0923394	-0.1302072	0.2317564	0.5824087	0.4077671
0.52	0.0511455	0.0914743	-0.1281408	0.2304317	0.5760769	0.4080540
0.53	0.0515169	0.0905982	-0.1260522	0.2290861	0.5696065	0.4083356
0.54	0.0518890	0.0897107	-0.1239408	0.2277188	0.5629920	0.4086117
0.55	0.0522616	0.0888116	-0.1218060	0.2263292	0.5562272	0.4088819
0.56	0.0526348	0.0879005	-0.1196469	0.2249166	0.5493061	0.4091462
0.57	0.0530086	0.0869769	-0.1174629	0.2234802	0.5422219	0.4094041
0.58	0.0533830	0.0860405	-0.1152532	0.2220192	0.5349676	0.4096555
0.59	0.0537580	0.0850907	-0.1130168	0.2205327	0.5275357	0.4099000
0.60	0.0541335	0.0841273	-0.1107529	0.2190199	0.5199182	0.4101373
0.61	0.0545096	0.0831496	-0.1084606	0.2174798	0.5121068	0.4103671
0.62	0.0548863	0.0821572	-0.1061388	0.2159113	0.5040925	0.4105890
0.63	0.0552635	0.0811494	-0.1037864	0.2143134	0.4958656	0.4108027
0.64	0.0556413	0.0801258	-0.1014024	0.2126850	0.4874161	0.4110078
0.65	0.0560196	0.0790856	-0.0989854	0.2110247	0.4787332	0.4112038
0.66	0.0563985	0.0780283	-0.0965341	0.2093312	0.4698052	0.4113902
0.67	0.0567780	0.0769531	-0.0940472	0.2076032	0.4606199	0.4115667
0.68	0.0571580	0.0758591	-0.0915231	0.2058391	0.4511642	0.4117326
0.69	0.0575385	0.0747456	-0.0889602	0.2040373	0.4414239	0.4118874
0.70	0.0579196	0.0736118	-0.0863568	0.2021960	0.4313840	0.4120306
0.71	0.0583012	0.0724565	-0.0837109	0.2003133	0.4210286	0.4121614
0.72	0.0586834	0.0712788	-0.0810204	0.1983872	0.4103405	0.4122793
0.73	0.0590661	0.0700774	-0.0782831	0.1964154	0.3993014	0.4123834
0.74	0.0594494	0.0688513	-0.0754966	0.1943954	0.3878919	0.4124729
0.75	0.0598332	0.0675989	-0.0726581	0.1923246	0.3760910	0.4125470

Code

summary(sensitivity_moderator) 

      ri_t             beta               se              ci.lb         
 Min.   :0.5000   Min.   :0.05040   Min.   :0.06760   Min.   :-0.13225  
 1st Qu.:0.5625   1st Qu.:0.05273   1st Qu.:0.07502   1st Qu.:-0.11910  
 Median :0.6250   Median :0.05507   Median :0.08165   Median :-0.10496  
 Mean   :0.6250   Mean   :0.05509   Mean   :0.08121   Mean   :-0.10407  
 3rd Qu.:0.6875   3rd Qu.:0.05744   3rd Qu.:0.08767   3rd Qu.:-0.08960  
 Max.   :0.7500   Max.   :0.05983   Max.   :0.09319   Max.   :-0.07266  
     ci.ub            pvalue           sigma2      
 Min.   :0.1923   Min.   :0.3761   Min.   :0.4075  
 1st Qu.:0.2045   1st Qu.:0.4439   1st Qu.:0.4092  
 Median :0.2151   Median :0.5000   Median :0.4107  
 Mean   :0.2143   Mean   :0.4937   Mean   :0.4105  
 3rd Qu.:0.2246   3rd Qu.:0.5475   3rd Qu.:0.4118  
 Max.   :0.2331   Max.   :0.5886   Max.   :0.4125  

Multiple representation formats

Code

#mono vs. multiple representation
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$rep_nr <- as.factor(dataN33_prepost$rep_nr)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = rep_nr)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.3057128	0.2115950	-0.1090059	0.7204314	0.1485137	0.3646199
0.51	0.3039749	0.2107303	-0.1090488	0.7169986	0.1491661	0.3649269
0.52	0.3021921	0.2098480	-0.1091025	0.7134866	0.1498527	0.3652316
0.53	0.3003626	0.2089478	-0.1091675	0.7098927	0.1505757	0.3655340
0.54	0.2984847	0.2080290	-0.1092446	0.7062139	0.1513377	0.3658341
0.55	0.2965562	0.2070910	-0.1093348	0.7024471	0.1521413	0.3661320
0.56	0.2945751	0.2061333	-0.1094388	0.6985890	0.1529895	0.3664276
0.57	0.2925392	0.2051553	-0.1095577	0.6946362	0.1538855	0.3667210
0.58	0.2904462	0.2041561	-0.1096924	0.6905848	0.1548327	0.3670122
0.59	0.2882936	0.2031352	-0.1098441	0.6864312	0.1558348	0.3673012
0.60	0.2860787	0.2020917	-0.1100138	0.6821713	0.1568960	0.3675882
0.61	0.2837989	0.2010250	-0.1102028	0.6778006	0.1580208	0.3678731
0.62	0.2814511	0.1999341	-0.1104125	0.6733147	0.1592141	0.3681560
0.63	0.2790323	0.1988182	-0.1106441	0.6687088	0.1604812	0.3684371
0.64	0.2765392	0.1976764	-0.1108993	0.6639777	0.1618282	0.3687164
0.65	0.2739682	0.1965076	-0.1111797	0.6591161	0.1632614	0.3689941
0.66	0.2713156	0.1953110	-0.1114869	0.6541180	0.1647882	0.3692702
0.67	0.2685774	0.1940853	-0.1118228	0.6489775	0.1664163	0.3695450
0.68	0.2657493	0.1928294	-0.1121894	0.6436880	0.1681547	0.3698186
0.69	0.2628269	0.1915422	-0.1125889	0.6382426	0.1700130	0.3700912
0.70	0.2598052	0.1902222	-0.1130234	0.6326338	0.1720022	0.3703631
0.71	0.2566791	0.1888681	-0.1134955	0.6268537	0.1741342	0.3706346
0.72	0.2534430	0.1874783	-0.1140078	0.6208938	0.1764226	0.3709059
0.73	0.2500910	0.1860514	-0.1145631	0.6147451	0.1788826	0.3711774
0.74	0.2466167	0.1845856	-0.1151645	0.6083978	0.1815311	0.3714496
0.75	0.2430132	0.1830791	-0.1158153	0.6018416	0.1843873	0.3717228

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.2430   Min.   :0.1831   Min.   :-0.1158  
 1st Qu.:0.5625   1st Qu.:0.2636   1st Qu.:0.1919   1st Qu.:-0.1125  
 Median :0.6250   Median :0.2802   Median :0.1994   Median :-0.1105  
 Mean   :0.6250   Mean   :0.2782   Mean   :0.1987   Mean   :-0.1112  
 3rd Qu.:0.6875   3rd Qu.:0.2941   3rd Qu.:0.2059   3rd Qu.:-0.1095  
 Max.   :0.7500   Max.   :0.3057   Max.   :0.2116   Max.   :-0.1090  
     ci.ub            pvalue           sigma2      
 Min.   :0.6018   Min.   :0.1485   Min.   :0.3646  
 1st Qu.:0.6396   1st Qu.:0.1532   1st Qu.:0.3665  
 Median :0.6710   Median :0.1598   Median :0.3683  
 Mean   :0.6675   Mean   :0.1621   Mean   :0.3682  
 3rd Qu.:0.6976   3rd Qu.:0.1695   3rd Qu.:0.3700  
 Max.   :0.7204   Max.   :0.1844   Max.   :0.3717  

Level of feedback

Code

## LEVEL ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$order<- as.factor(dataN33_prepost$FB_order)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = order)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.2663898	0.3260957	-0.3727460	0.9055255	0.4139817	0.4149546
0.51	0.2663955	0.3261140	-0.3727761	0.9055671	0.4139978	0.4152738
0.52	0.2664048	0.3261300	-0.3727982	0.9056078	0.4140045	0.4155871
0.53	0.2664176	0.3261437	-0.3728122	0.9056475	0.4140016	0.4158945
0.54	0.2664342	0.3261549	-0.3728176	0.9056861	0.4139886	0.4161955
0.55	0.2664548	0.3261636	-0.3728142	0.9057237	0.4139651	0.4164901
0.56	0.2664793	0.3261697	-0.3728016	0.9057602	0.4139307	0.4167779
0.57	0.2665082	0.3261730	-0.3727793	0.9057956	0.4138850	0.4170586
0.58	0.2665414	0.3261735	-0.3727469	0.9058297	0.4138275	0.4173318
0.59	0.2665792	0.3261710	-0.3727042	0.9058626	0.4137576	0.4175974
0.60	0.2666218	0.3261653	-0.3726505	0.9058941	0.4136749	0.4178548
0.61	0.2666695	0.3261564	-0.3725854	0.9059243	0.4135787	0.4181038
0.62	0.2667224	0.3261441	-0.3725083	0.9059530	0.4134684	0.4183440
0.63	0.2667808	0.3261282	-0.3724187	0.9059802	0.4133433	0.4185748
0.64	0.2668449	0.3261085	-0.3723160	0.9060059	0.4132029	0.4187959
0.65	0.2669151	0.3260849	-0.3721996	0.9060299	0.4130462	0.4190069
0.66	0.2669917	0.3260573	-0.3720688	0.9060522	0.4128725	0.4192071
0.67	0.2670750	0.3260252	-0.3719227	0.9060727	0.4126809	0.4193960
0.68	0.2671653	0.3259886	-0.3717607	0.9060913	0.4124704	0.4195730
0.69	0.2672631	0.3259473	-0.3715818	0.9061080	0.4122400	0.4197375
0.70	0.2673687	0.3259008	-0.3713851	0.9061226	0.4119886	0.4198889
0.71	0.2674827	0.3258490	-0.3711697	0.9061351	0.4117150	0.4200263
0.72	0.2676056	0.3257916	-0.3709343	0.9061455	0.4114178	0.4201490
0.73	0.2677379	0.3257282	-0.3706777	0.9061536	0.4110956	0.4202560
0.74	0.2678803	0.3256585	-0.3703987	0.9061593	0.4107467	0.4203466
0.75	0.2680335	0.3255821	-0.3700957	0.9061627	0.4103693	0.4204195

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.2664   Min.   :0.3256   Min.   :-0.3728  
 1st Qu.:0.5625   1st Qu.:0.2665   1st Qu.:0.3260   1st Qu.:-0.3728  
 Median :0.6250   Median :0.2668   Median :0.3261   Median :-0.3725  
 Mean   :0.6250   Mean   :0.2669   Mean   :0.3260   Mean   :-0.3721  
 3rd Qu.:0.6875   3rd Qu.:0.2672   3rd Qu.:0.3262   3rd Qu.:-0.3716  
 Max.   :0.7500   Max.   :0.2680   Max.   :0.3262   Max.   :-0.3701  
     ci.ub            pvalue           sigma2      
 Min.   :0.9055   Min.   :0.4104   Min.   :0.4150  
 1st Qu.:0.9058   1st Qu.:0.4123   1st Qu.:0.4168  
 Median :0.9060   Median :0.4134   Median :0.4185  
 Mean   :0.9059   Mean   :0.4130   Mean   :0.4182  
 3rd Qu.:0.9061   3rd Qu.:0.4139   3rd Qu.:0.4197  
 Max.   :0.9062   Max.   :0.4140   Max.   :0.4204  

Higher level only

Code

## HIGHER LEVEL ONLY ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$order_hi <- as.factor(ifelse(dataN33_prepost$FB_order == 2, 1, 0))
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = order_hi)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.2663898	0.3260957	-0.9055256	0.3727460	0.4139817	0.4149546
0.51	-0.2663955	0.3261140	-0.9055671	0.3727761	0.4139978	0.4152738
0.52	-0.2664048	0.3261300	-0.9056078	0.3727982	0.4140045	0.4155871
0.53	-0.2664176	0.3261437	-0.9056475	0.3728122	0.4140016	0.4158945
0.54	-0.2664343	0.3261549	-0.9056862	0.3728177	0.4139886	0.4161955
0.55	-0.2664548	0.3261636	-0.9057237	0.3728142	0.4139651	0.4164901
0.56	-0.2664793	0.3261697	-0.9057602	0.3728016	0.4139307	0.4167779
0.57	-0.2665082	0.3261730	-0.9057956	0.3727793	0.4138850	0.4170586
0.58	-0.2665414	0.3261735	-0.9058297	0.3727469	0.4138275	0.4173318
0.59	-0.2665792	0.3261710	-0.9058626	0.3727042	0.4137576	0.4175974
0.60	-0.2666218	0.3261653	-0.9058941	0.3726505	0.4136749	0.4178548
0.61	-0.2666695	0.3261564	-0.9059243	0.3725853	0.4135786	0.4181038
0.62	-0.2667224	0.3261441	-0.9059530	0.3725083	0.4134684	0.4183440
0.63	-0.2667808	0.3261282	-0.9059802	0.3724187	0.4133433	0.4185748
0.64	-0.2668449	0.3261085	-0.9060059	0.3723160	0.4132029	0.4187960
0.65	-0.2669151	0.3260849	-0.9060299	0.3721996	0.4130462	0.4190069
0.66	-0.2669917	0.3260572	-0.9060522	0.3720688	0.4128725	0.4192070
0.67	-0.2670750	0.3260252	-0.9060727	0.3719227	0.4126809	0.4193960
0.68	-0.2671653	0.3259886	-0.9060913	0.3717607	0.4124704	0.4195730
0.69	-0.2672631	0.3259473	-0.9061080	0.3715818	0.4122400	0.4197375
0.70	-0.2673687	0.3259008	-0.9061226	0.3713851	0.4119886	0.4198889
0.71	-0.2674827	0.3258491	-0.9061351	0.3711697	0.4117150	0.4200263
0.72	-0.2676056	0.3257916	-0.9061455	0.3709342	0.4114178	0.4201489
0.73	-0.2677379	0.3257282	-0.9061536	0.3706777	0.4110956	0.4202560
0.74	-0.2678803	0.3256585	-0.9061593	0.3703987	0.4107467	0.4203465
0.75	-0.2680335	0.3255821	-0.9061627	0.3700957	0.4103693	0.4204195

Code

summary(sensitivity_moderator) 

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :-0.2680   Min.   :0.3256   Min.   :-0.9062  
 1st Qu.:0.5625   1st Qu.:-0.2672   1st Qu.:0.3260   1st Qu.:-0.9061  
 Median :0.6250   Median :-0.2668   Median :0.3261   Median :-0.9060  
 Mean   :0.6250   Mean   :-0.2669   Mean   :0.3260   Mean   :-0.9059  
 3rd Qu.:0.6875   3rd Qu.:-0.2665   3rd Qu.:0.3262   3rd Qu.:-0.9058  
 Max.   :0.7500   Max.   :-0.2664   Max.   :0.3262   Max.   :-0.9055  
     ci.ub            pvalue           sigma2      
 Min.   :0.3701   Min.   :0.4104   Min.   :0.4150  
 1st Qu.:0.3716   1st Qu.:0.4123   1st Qu.:0.4168  
 Median :0.3725   Median :0.4134   Median :0.4185  
 Mean   :0.3721   Mean   :0.4130   Mean   :0.4182  
 3rd Qu.:0.3728   3rd Qu.:0.4139   3rd Qu.:0.4197  
 Max.   :0.3728   Max.   :0.4140   Max.   :0.4204  

####Both lower and higher level

Code

## BOTH LOWER AND HIGHER LEVEL ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$order_hilow <- ifelse(dataN33_prepost$FB_order == 3, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = order_hilow)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.2663898	0.3260957	-0.3727460	0.9055255	0.4139817	0.4149546
0.51	0.2663955	0.3261140	-0.3727761	0.9055671	0.4139978	0.4152738
0.52	0.2664048	0.3261300	-0.3727982	0.9056078	0.4140045	0.4155871
0.53	0.2664176	0.3261437	-0.3728122	0.9056475	0.4140016	0.4158945
0.54	0.2664342	0.3261549	-0.3728176	0.9056861	0.4139886	0.4161955
0.55	0.2664548	0.3261636	-0.3728142	0.9057237	0.4139651	0.4164901
0.56	0.2664793	0.3261697	-0.3728016	0.9057602	0.4139307	0.4167779
0.57	0.2665082	0.3261730	-0.3727793	0.9057956	0.4138850	0.4170586
0.58	0.2665414	0.3261735	-0.3727469	0.9058297	0.4138275	0.4173318
0.59	0.2665792	0.3261710	-0.3727042	0.9058626	0.4137576	0.4175974
0.60	0.2666218	0.3261653	-0.3726505	0.9058941	0.4136749	0.4178548
0.61	0.2666695	0.3261564	-0.3725853	0.9059243	0.4135786	0.4181038
0.62	0.2667224	0.3261441	-0.3725083	0.9059530	0.4134684	0.4183440
0.63	0.2667808	0.3261282	-0.3724187	0.9059802	0.4133433	0.4185748
0.64	0.2668449	0.3261085	-0.3723160	0.9060059	0.4132029	0.4187959
0.65	0.2669151	0.3260849	-0.3721996	0.9060299	0.4130462	0.4190069
0.66	0.2669917	0.3260572	-0.3720688	0.9060522	0.4128725	0.4192070
0.67	0.2670750	0.3260252	-0.3719227	0.9060727	0.4126809	0.4193960
0.68	0.2671653	0.3259886	-0.3717607	0.9060913	0.4124704	0.4195730
0.69	0.2672631	0.3259473	-0.3715818	0.9061080	0.4122400	0.4197375
0.70	0.2673687	0.3259008	-0.3713851	0.9061226	0.4119886	0.4198889
0.71	0.2674827	0.3258490	-0.3711697	0.9061351	0.4117150	0.4200263
0.72	0.2676056	0.3257916	-0.3709343	0.9061455	0.4114178	0.4201489
0.73	0.2677379	0.3257282	-0.3706777	0.9061536	0.4110956	0.4202560
0.74	0.2678803	0.3256585	-0.3703987	0.9061593	0.4107467	0.4203465
0.75	0.2680335	0.3255821	-0.3700957	0.9061627	0.4103693	0.4204195

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.2664   Min.   :0.3256   Min.   :-0.3728  
 1st Qu.:0.5625   1st Qu.:0.2665   1st Qu.:0.3260   1st Qu.:-0.3728  
 Median :0.6250   Median :0.2668   Median :0.3261   Median :-0.3725  
 Mean   :0.6250   Mean   :0.2669   Mean   :0.3260   Mean   :-0.3721  
 3rd Qu.:0.6875   3rd Qu.:0.2672   3rd Qu.:0.3262   3rd Qu.:-0.3716  
 Max.   :0.7500   Max.   :0.2680   Max.   :0.3262   Max.   :-0.3701  
     ci.ub            pvalue           sigma2      
 Min.   :0.9055   Min.   :0.4104   Min.   :0.4150  
 1st Qu.:0.9058   1st Qu.:0.4123   1st Qu.:0.4168  
 Median :0.9060   Median :0.4134   Median :0.4185  
 Mean   :0.9059   Mean   :0.4130   Mean   :0.4182  
 3rd Qu.:0.9061   3rd Qu.:0.4139   3rd Qu.:0.4197  
 Max.   :0.9062   Max.   :0.4140   Max.   :0.4204  

Specificity

Code

## SPECIFICITY ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$spec <- as.factor(dataN33_prepost$FB_specificity)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = spec)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.3986597	0.4950525	-0.5716252	1.368945	0.4206536	0.4138367
0.51	0.3985402	0.4950218	-0.5716848	1.368765	0.4207641	0.4141115
0.52	0.3984099	0.4949888	-0.5717502	1.368570	0.4208850	0.4143823
0.53	0.3982683	0.4949531	-0.5718220	1.368359	0.4210167	0.4146487
0.54	0.3981147	0.4949148	-0.5719005	1.368130	0.4211599	0.4149107
0.55	0.3979486	0.4948737	-0.5719861	1.367883	0.4213151	0.4151680
0.56	0.3977694	0.4948301	-0.5720797	1.367619	0.4214833	0.4154209
0.57	0.3975762	0.4947831	-0.5721808	1.367333	0.4216648	0.4156683
0.58	0.3973684	0.4947330	-0.5722905	1.367027	0.4218605	0.4159105
0.59	0.3971451	0.4946797	-0.5724093	1.366700	0.4220713	0.4161472
0.60	0.3969055	0.4946230	-0.5725378	1.366349	0.4222982	0.4163782
0.61	0.3966486	0.4945628	-0.5726766	1.365974	0.4225421	0.4166033
0.62	0.3963734	0.4944989	-0.5728266	1.365573	0.4228042	0.4168221
0.63	0.3960787	0.4944311	-0.5729885	1.365146	0.4230856	0.4170345
0.64	0.3957633	0.4943594	-0.5731633	1.364690	0.4233877	0.4172401
0.65	0.3954259	0.4942834	-0.5733518	1.364204	0.4237118	0.4174386
0.66	0.3950651	0.4942031	-0.5735552	1.363685	0.4240595	0.4176297
0.67	0.3946792	0.4941182	-0.5737747	1.363133	0.4244327	0.4178130
0.68	0.3942666	0.4940286	-0.5740116	1.362545	0.4248331	0.4179883
0.69	0.3938253	0.4939339	-0.5742673	1.361918	0.4252629	0.4181550
0.70	0.3933532	0.4938339	-0.5745435	1.361250	0.4257244	0.4183128
0.71	0.3928480	0.4937284	-0.5748420	1.360538	0.4262202	0.4184613
0.72	0.3923071	0.4936172	-0.5751648	1.359779	0.4267533	0.4186000
0.73	0.3917275	0.4934999	-0.5755144	1.358969	0.4273268	0.4187283
0.74	0.3911062	0.4933761	-0.5758932	1.358106	0.4279444	0.4188459
0.75	0.3904395	0.4932457	-0.5763043	1.357183	0.4286102	0.4189520

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.3904   Min.   :0.4932   Min.   :-0.5763  
 1st Qu.:0.5625   1st Qu.:0.3939   1st Qu.:0.4940   1st Qu.:-0.5742  
 Median :0.6250   Median :0.3962   Median :0.4945   Median :-0.5729  
 Mean   :0.6250   Mean   :0.3956   Mean   :0.4944   Mean   :-0.5733  
 3rd Qu.:0.6875   3rd Qu.:0.3977   3rd Qu.:0.4948   3rd Qu.:-0.5721  
 Max.   :0.7500   Max.   :0.3987   Max.   :0.4951   Max.   :-0.5716  
     ci.ub           pvalue           sigma2      
 Min.   :1.357   Min.   :0.4207   Min.   :0.4138  
 1st Qu.:1.362   1st Qu.:0.4215   1st Qu.:0.4155  
 Median :1.365   Median :0.4229   Median :0.4169  
 Mean   :1.365   Mean   :0.4235   Mean   :0.4167  
 3rd Qu.:1.368   3rd Qu.:0.4252   3rd Qu.:0.4181  
 Max.   :1.369   Max.   :0.4286   Max.   :0.4190  

Tool numbers

Code

## TOOL NUMBERS ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$tool <- as.factor(dataN33_prepost$FB_tool_numbers)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = tool)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.1242885	0.0637999	-0.2493340	0.0007571	0.0514033	0.3321570
0.51	-0.1242588	0.0638123	-0.2493286	0.0008111	0.0515043	0.3325097
0.52	-0.1242277	0.0638244	-0.2493212	0.0008658	0.0516069	0.3328590
0.53	-0.1241952	0.0638361	-0.2493117	0.0009213	0.0517110	0.3332048
0.54	-0.1241613	0.0638475	-0.2493000	0.0009775	0.0518166	0.3335469
0.55	-0.1241258	0.0638584	-0.2492860	0.0010344	0.0519238	0.3338851
0.56	-0.1240888	0.0638690	-0.2492697	0.0010922	0.0520326	0.3342192
0.57	-0.1240502	0.0638791	-0.2492510	0.0011507	0.0521431	0.3345490
0.58	-0.1240099	0.0638889	-0.2492297	0.0012100	0.0522552	0.3348744
0.59	-0.1239678	0.0638981	-0.2492058	0.0012701	0.0523692	0.3351949
0.60	-0.1239240	0.0639069	-0.2491792	0.0013311	0.0524849	0.3355105
0.61	-0.1238784	0.0639151	-0.2491496	0.0013929	0.0526024	0.3358208
0.62	-0.1238308	0.0639228	-0.2491172	0.0014556	0.0527218	0.3361255
0.63	-0.1237812	0.0639299	-0.2490816	0.0015191	0.0528430	0.3364243
0.64	-0.1237296	0.0639365	-0.2490428	0.0015836	0.0529662	0.3367169
0.65	-0.1236759	0.0639424	-0.2490006	0.0016489	0.0530914	0.3370029
0.66	-0.1236199	0.0639476	-0.2489549	0.0017151	0.0532185	0.3372819
0.67	-0.1235617	0.0639521	-0.2489055	0.0017822	0.0533476	0.3375535
0.68	-0.1235011	0.0639559	-0.2488523	0.0018501	0.0534786	0.3378172
0.69	-0.1234380	0.0639588	-0.2487950	0.0019190	0.0536117	0.3380725
0.70	-0.1233724	0.0639609	-0.2487336	0.0019887	0.0537468	0.3383188
0.71	-0.1233042	0.0639621	-0.2486677	0.0020593	0.0538839	0.3385556
0.72	-0.1232333	0.0639624	-0.2485973	0.0021306	0.0540228	0.3387822
0.73	-0.1231596	0.0639616	-0.2485220	0.0022027	0.0541636	0.3389979
0.74	-0.1230831	0.0639597	-0.2484417	0.0022756	0.0543062	0.3392019
0.75	-0.1230035	0.0639565	-0.2483560	0.0023490	0.0544504	0.3393933

Code

summary(sensitivity_moderator)

      ri_t             beta               se              ci.lb        
 Min.   :0.5000   Min.   :-0.1243   Min.   :0.06380   Min.   :-0.2493  
 1st Qu.:0.5625   1st Qu.:-0.1241   1st Qu.:0.06387   1st Qu.:-0.2493  
 Median :0.6250   Median :-0.1238   Median :0.06393   Median :-0.2491  
 Mean   :0.6250   Mean   :-0.1237   Mean   :0.06391   Mean   :-0.2490  
 3rd Qu.:0.6875   3rd Qu.:-0.1235   3rd Qu.:0.06396   3rd Qu.:-0.2488  
 Max.   :0.7500   Max.   :-0.1230   Max.   :0.06396   Max.   :-0.2484  
     ci.ub               pvalue            sigma2      
 Min.   :0.0007571   Min.   :0.05140   Min.   :0.3322  
 1st Qu.:0.0011068   1st Qu.:0.05206   1st Qu.:0.3343  
 Median :0.0014874   Median :0.05278   Median :0.3363  
 Mean   :0.0015113   Mean   :0.05283   Mean   :0.3361  
 3rd Qu.:0.0019018   3rd Qu.:0.05358   3rd Qu.:0.3380  
 Max.   :0.0023490   Max.   :0.05445   Max.   :0.3394  

Code

#post-hoc:
CohViz <- rma(yi, vi, data=data_pp_es, subset = tool =="1") 
CohViz

Random-Effects Model (k = 12; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0.2166 (SE = 0.1025)
tau (square root of estimated tau^2 value):      0.4654
I^2 (total heterogeneity / total variability):   95.09%
H^2 (total variability / sampling variability):  20.36

Test for Heterogeneity:
Q(df = 11) = 92.3366, p-val < .0001

Model Results:

estimate      se    zval    pval   ci.lb   ci.ub      
  0.5172  0.1418  3.6481  0.0003  0.2393  0.7951  *** 

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Code

criterion <- rma(yi, vi, data=data_pp_es, subset = tool =="2") 
criterion

Random-Effects Model (k = 4; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 2.8417 (SE = 2.4227)
tau (square root of estimated tau^2 value):      1.6857
I^2 (total heterogeneity / total variability):   97.59%
H^2 (total variability / sampling variability):  41.47

Test for Heterogeneity:
Q(df = 3) = 72.5270, p-val < .0001

Model Results:

estimate      se    zval    pval    ci.lb   ci.ub    
  1.3118  0.8613  1.5231  0.1277  -0.3763  2.9999    

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Code

writingPal <- rma(yi, vi, data=data_pp_es, subset = tool =="3") 
writingPal

Random-Effects Model (k = 6; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0 (SE = 0.0061)
tau (square root of estimated tau^2 value):      0
I^2 (total heterogeneity / total variability):   0.00%
H^2 (total variability / sampling variability):  1.00

Test for Heterogeneity:
Q(df = 5) = 1.3625, p-val = 0.9284

Model Results:

estimate      se    zval    pval   ci.lb   ci.ub      
  0.2042  0.0403  5.0725  <.0001  0.1253  0.2831  *** 

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Code

NCWrite <- rma(yi, vi, data=data_pp_es, subset = tool =="4") 
NCWrite

Random-Effects Model (k = 2; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0.1371 (SE = 0.1978)
tau (square root of estimated tau^2 value):      0.3703
I^2 (total heterogeneity / total variability):   98.00%
H^2 (total variability / sampling variability):  50.10

Test for Heterogeneity:
Q(df = 1) = 50.1046, p-val < .0001

Model Results:

estimate      se    zval    pval   ci.lb   ci.ub     
  0.7057  0.2645  2.6684  0.0076  0.1873  1.2240  ** 

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Code

eRevise <- rma(yi, vi, data=data_pp_es, subset = tool =="5") 
eRevise

Random-Effects Model (k = 2; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0.0331 (SE = 0.0509)
tau (square root of estimated tau^2 value):      0.1820
I^2 (total heterogeneity / total variability):   91.98%
H^2 (total variability / sampling variability):  12.47

Test for Heterogeneity:
Q(df = 1) = 12.4698, p-val = 0.0004

Model Results:

estimate      se    zval    pval   ci.lb   ci.ub    
  0.2825  0.1341  2.1061  0.0352  0.0196  0.5454  * 

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Code

cRaterML <- rma(yi, vi, data=data_pp_es, subset = tool =="6") 
cRaterML

Random-Effects Model (k = 1; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0
tau (square root of estimated tau^2 value):      0
I^2 (total heterogeneity / total variability):   0.00%
H^2 (total variability / sampling variability):  1.00

Test for Heterogeneity:
Q(df = 0) = 0.0000, p-val = 1.0000

Model Results:

estimate      se    zval    pval   ci.lb   ci.ub      
  0.3517  0.0631  5.5714  <.0001  0.2280  0.4754  *** 

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Code

MIWrite <- rma(yi, vi, data=data_pp_es, subset = tool =="7")
MIWrite

Random-Effects Model (k = 5; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0.1333 (SE = 0.1000)
tau (square root of estimated tau^2 value):      0.3651
I^2 (total heterogeneity / total variability):   96.71%
H^2 (total variability / sampling variability):  30.38

Test for Heterogeneity:
Q(df = 4) = 69.0614, p-val < .0001

Model Results:

estimate      se    zval    pval   ci.lb   ci.ub    
  0.3363  0.1682  1.9990  0.0456  0.0066  0.6659  * 

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Code

ChatGPT <- rma(yi, vi, data=data_pp_es, subset = tool =="8") 
ChatGPT

Random-Effects Model (k = 1; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0
tau (square root of estimated tau^2 value):      0
I^2 (total heterogeneity / total variability):   0.00%
H^2 (total variability / sampling variability):  1.00

Test for Heterogeneity:
Q(df = 0) = 0.0000, p-val = 1.0000

Model Results:

estimate      se      zval    pval    ci.lb    ci.ub      
 -0.6650  0.0490  -13.5637  <.0001  -0.7611  -0.5689  *** 

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Code

dat.comp <- data.frame(estimate = c(coef(CohViz), coef(criterion), coef(writingPal), coef(NCWrite),coef(eRevise), coef(cRaterML), coef(MIWrite), coef(ChatGPT)), stderror = c(CohViz$se, criterion$se, writingPal$se, NCWrite$se, eRevise$se, cRaterML$se, MIWrite$se, ChatGPT$se), meta = c("1","2", "3", "4", "5", "6", "7", "8"), tau2 = round(c(CohViz$tau2, criterion$tau2, writingPal$tau2, NCWrite$tau2, eRevise$tau2, cRaterML$tau2, MIWrite$tau2, ChatGPT$tau2),3))

dat.comp 

    estimate   stderror meta  tau2
1  0.5172196 0.14177941    1 0.217
2  1.3117891 0.86127386    2 2.842
3  0.2042270 0.04026152    3 0.000
4  0.7056804 0.26446012    4 0.137
5  0.2824778 0.13412655    5 0.033
6  0.3516930 0.06312455    6 0.000
7  0.3362509 0.16820946    7 0.133
8 -0.6649928 0.04902747    8 0.000

Code

rma(estimate, sei=stderror, mods = ~ meta, method="FE", data=dat.comp, digits=3) 

Fixed-Effects with Moderators Model (k = 8)

I^2 (residual heterogeneity / unaccounted variability): 0.00%
H^2 (unaccounted variability / sampling variability):   1.00
R^2 (amount of heterogeneity accounted for):            NA%

Test for Residual Heterogeneity:
QE(df = 0) = 0.000, p-val = 1.000

Test of Moderators (coefficients 2:8):
QM(df = 7) = 271.919, p-val < .001

Model Results:

         estimate     se    zval   pval   ci.lb   ci.ub      
intrcpt     0.517  0.142   3.648  <.001   0.239   0.795  *** 
meta2       0.795  0.873   0.910  0.363  -0.916   2.505      
meta3      -0.313  0.147  -2.124  0.034  -0.602  -0.024    * 
meta4       0.188  0.300   0.628  0.530  -0.400   0.777      
meta5      -0.235  0.195  -1.203  0.229  -0.617   0.148      
meta6      -0.166  0.155  -1.067  0.286  -0.470   0.139      
meta7      -0.181  0.220  -0.823  0.411  -0.612   0.250      
meta8      -1.182  0.150  -7.881  <.001  -1.476  -0.888  *** 

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Code

data_pp_es$tool <- dataN33_prepost$FB_tool_numbers
res <- rma(yi, vi, mods = ~ factor(tool), data=data_pp_es)
res

Mixed-Effects Model (k = 33; tau^2 estimator: REML)

tau^2 (estimated amount of residual heterogeneity):     0.3365 (SE = 0.1021)
tau (square root of estimated tau^2 value):             0.5801
I^2 (residual heterogeneity / unaccounted variability): 97.68%
H^2 (unaccounted variability / sampling variability):   43.04
R^2 (amount of heterogeneity accounted for):            0.00%

Test for Residual Heterogeneity:
QE(df = 25) = 297.8620, p-val < .0001

Test of Moderators (coefficients 2:8):
QM(df = 7) = 9.0746, p-val = 0.2473

Model Results:

               estimate      se     zval    pval    ci.lb    ci.ub     
intrcpt          0.5311  0.1737   3.0585  0.0022   0.1908   0.8715  ** 
factor(tool)2    0.5331  0.3761   1.4173  0.1564  -0.2041   1.2703     
factor(tool)3   -0.3287  0.2965  -1.1083  0.2677  -0.9099   0.2525     
factor(tool)4    0.1752  0.4470   0.3920  0.6951  -0.7009   1.0513     
factor(tool)5   -0.2509  0.4471  -0.5613  0.5746  -1.1271   0.6253     
factor(tool)6   -0.1794  0.6088  -0.2947  0.7682  -1.3727   1.0138     
factor(tool)7   -0.1855  0.3149  -0.5892  0.5558  -0.8026   0.4316     
factor(tool)8   -1.1961  0.6075  -1.9688  0.0490  -2.3868  -0.0054   * 

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Prior knowledge

Code

## PRIOR KNOWLEDGE ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$WQP_pre <- as.factor(dataN33_prepost$WQP_pre)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = WQP_pre)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.0161422	0.0049863	-0.0259152	-0.0063692	0.0012066	0.3961016
0.51	-0.0161278	0.0049420	-0.0258139	-0.0064417	0.0011007	0.3964070
0.52	-0.0161124	0.0048971	-0.0257105	-0.0065143	0.0010012	0.3967075
0.53	-0.0160960	0.0048516	-0.0256050	-0.0065870	0.0009078	0.3970029
0.54	-0.0160785	0.0048056	-0.0254972	-0.0066598	0.0008204	0.3972929
0.55	-0.0160599	0.0047589	-0.0253871	-0.0067326	0.0007389	0.3975774
0.56	-0.0160400	0.0047116	-0.0252745	-0.0068055	0.0006631	0.3978559
0.57	-0.0160189	0.0046636	-0.0251594	-0.0068785	0.0005928	0.3981283
0.58	-0.0159965	0.0046150	-0.0250417	-0.0069514	0.0005278	0.3983943
0.59	-0.0159726	0.0045656	-0.0249211	-0.0070242	0.0004680	0.3986535
0.60	-0.0159473	0.0045156	-0.0247976	-0.0070969	0.0004130	0.3989056
0.61	-0.0159203	0.0044648	-0.0246711	-0.0071695	0.0003628	0.3991503
0.62	-0.0158916	0.0044132	-0.0245413	-0.0072420	0.0003171	0.3993870
0.63	-0.0158611	0.0043608	-0.0244081	-0.0073141	0.0002756	0.3996156
0.64	-0.0158287	0.0043076	-0.0242713	-0.0073860	0.0002382	0.3998355
0.65	-0.0157941	0.0042535	-0.0241308	-0.0074575	0.0002046	0.4000462
0.66	-0.0157574	0.0041985	-0.0239862	-0.0075286	0.0001746	0.4002472
0.67	-0.0157183	0.0041425	-0.0238374	-0.0075991	0.0001480	0.4004380
0.68	-0.0156766	0.0040856	-0.0236842	-0.0076690	0.0001245	0.4006179
0.69	-0.0156322	0.0040276	-0.0235261	-0.0077382	0.0001039	0.4007865
0.70	-0.0155848	0.0039686	-0.0233631	-0.0078065	0.0000860	0.4009428
0.71	-0.0155343	0.0039084	-0.0231946	-0.0078739	0.0000705	0.4010863
0.72	-0.0154803	0.0038470	-0.0230203	-0.0079402	0.0000572	0.4012161
0.73	-0.0154226	0.0037844	-0.0228399	-0.0080052	0.0000460	0.4013314
0.74	-0.0153608	0.0037205	-0.0226528	-0.0080688	0.0000365	0.4014311
0.75	-0.0152946	0.0036552	-0.0224586	-0.0081307	0.0000286	0.4015142

Code

summary(sensitivity_moderator)

      ri_t             beta                se               ci.lb         
 Min.   :0.5000   Min.   :-0.01614   Min.   :0.003655   Min.   :-0.02592  
 1st Qu.:0.5625   1st Qu.:-0.01603   1st Qu.:0.004042   1st Qu.:-0.02525  
 Median :0.6250   Median :-0.01588   Median :0.004387   Median :-0.02447  
 Mean   :0.6250   Mean   :-0.01582   Mean   :0.004363   Mean   :-0.02437  
 3rd Qu.:0.6875   3rd Qu.:-0.01564   3rd Qu.:0.004700   3rd Qu.:-0.02357  
 Max.   :0.7500   Max.   :-0.01529   Max.   :0.004986   Max.   :-0.02246  
     ci.ub               pvalue              sigma2      
 Min.   :-0.008131   Min.   :2.859e-05   Min.   :0.3961  
 1st Qu.:-0.007721   1st Qu.:1.091e-04   1st Qu.:0.3979  
 Median :-0.007278   Median :2.963e-04   Median :0.3995  
 Mean   :-0.007269   Mean   :4.121e-04   Mean   :0.3993  
 3rd Qu.:-0.006824   3rd Qu.:6.455e-04   3rd Qu.:0.4007  
 Max.   :-0.006369   Max.   :1.207e-03   Max.   :0.4015  

Setting

Code

## SETTING ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$setting <- as.factor(dataN33_prepost$setting)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = setting)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.1725533	0.2739199	-0.7094264	0.3643199	0.5287334	0.4447521
0.51	-0.1754365	0.2733160	-0.7111260	0.3602530	0.5209501	0.4455022
0.52	-0.1783667	0.2726990	-0.7128468	0.3561135	0.5130609	0.4462607
0.53	-0.1813450	0.2720684	-0.7145893	0.3518993	0.5050648	0.4470280
0.54	-0.1843727	0.2714239	-0.7163538	0.3476084	0.4969607	0.4478045
0.55	-0.1874511	0.2707651	-0.7181408	0.3432387	0.4887479	0.4485904
0.56	-0.1905814	0.2700914	-0.7199508	0.3387879	0.4804255	0.4493860
0.57	-0.1937651	0.2694024	-0.7217841	0.3342539	0.4719929	0.4501919
0.58	-0.1970034	0.2686976	-0.7236411	0.3296343	0.4634494	0.4510082
0.59	-0.2002979	0.2679766	-0.7255224	0.3249266	0.4547947	0.4518355
0.60	-0.2036500	0.2672387	-0.7274283	0.3201282	0.4460285	0.4526742
0.61	-0.2070613	0.2664835	-0.7293593	0.3152367	0.4371506	0.4535245
0.62	-0.2105334	0.2657102	-0.7313158	0.3102491	0.4281612	0.4543870
0.63	-0.2140679	0.2649184	-0.7332983	0.3051626	0.4190606	0.4552622
0.64	-0.2176664	0.2641073	-0.7353072	0.2999744	0.4098492	0.4561504
0.65	-0.2213308	0.2632763	-0.7373429	0.2946813	0.4005279	0.4570523
0.66	-0.2250629	0.2624247	-0.7394058	0.2892801	0.3910977	0.4579682
0.67	-0.2288644	0.2615517	-0.7414964	0.2837676	0.3815600	0.4588988
0.68	-0.2327374	0.2606566	-0.7436150	0.2781401	0.3719166	0.4598445
0.69	-0.2366839	0.2597385	-0.7457620	0.2723943	0.3621694	0.4608060
0.70	-0.2407058	0.2587966	-0.7479378	0.2665262	0.3523211	0.4617837
0.71	-0.2448053	0.2578299	-0.7501426	0.2605320	0.3423746	0.4627784
0.72	-0.2489846	0.2568375	-0.7523769	0.2544077	0.3323334	0.4637905
0.73	-0.2532459	0.2558184	-0.7546408	0.2481490	0.3222015	0.4648207
0.74	-0.2575916	0.2547715	-0.7569345	0.2417514	0.3119834	0.4658698
0.75	-0.2620240	0.2536957	-0.7592584	0.2352105	0.3016845	0.4669382

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :-0.2620   Min.   :0.2537   Min.   :-0.7593  
 1st Qu.:0.5625   1st Qu.:-0.2357   1st Qu.:0.2600   1st Qu.:-0.7452  
 Median :0.6250   Median :-0.2123   Median :0.2653   Median :-0.7323  
 Mean   :0.6250   Mean   :-0.2141   Mean   :0.2648   Mean   :-0.7330  
 3rd Qu.:0.6875   3rd Qu.:-0.1914   3rd Qu.:0.2699   3rd Qu.:-0.7204  
 Max.   :0.7500   Max.   :-0.1726   Max.   :0.2739   Max.   :-0.7094  
     ci.ub            pvalue           sigma2      
 Min.   :0.2352   Min.   :0.3017   Min.   :0.4448  
 1st Qu.:0.2738   1st Qu.:0.3646   1st Qu.:0.4496  
 Median :0.3077   Median :0.4236   Median :0.4548  
 Mean   :0.3049   Mean   :0.4206   Mean   :0.4552  
 3rd Qu.:0.3377   3rd Qu.:0.4783   3rd Qu.:0.4606  
 Max.   :0.3643   Max.   :0.5287   Max.   :0.4669  

Educational level

Code

## EDUCATIONAL LEVEL (dummy-coded) ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$edu <- as.factor(dataN33_prepost$education_dummy)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = edu)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.4825145	0.3258198	-0.1560805	1.121110	0.1386267	0.3837417
0.51	0.4826074	0.3257584	-0.1558673	1.121082	0.1384763	0.3838775
0.52	0.4826918	0.3256956	-0.1556599	1.121043	0.1383314	0.3840102
0.53	0.4827671	0.3256313	-0.1554585	1.120993	0.1381920	0.3841396
0.54	0.4828332	0.3255655	-0.1552635	1.120930	0.1380585	0.3842656
0.55	0.4828896	0.3254981	-0.1550749	1.120854	0.1379308	0.3843880
0.56	0.4829361	0.3254290	-0.1548931	1.120765	0.1378094	0.3845066
0.57	0.4829720	0.3253582	-0.1547184	1.120662	0.1376945	0.3846213
0.58	0.4829971	0.3252856	-0.1545509	1.120545	0.1375861	0.3847318
0.59	0.4830110	0.3252111	-0.1543910	1.120413	0.1374848	0.3848382
0.60	0.4830130	0.3251346	-0.1542391	1.120265	0.1373906	0.3849400
0.61	0.4830027	0.3250560	-0.1540954	1.120101	0.1373040	0.3850371
0.62	0.4829797	0.3249753	-0.1539602	1.119920	0.1372251	0.3851293
0.63	0.4829432	0.3248923	-0.1538340	1.119720	0.1371544	0.3852163
0.64	0.4828928	0.3248070	-0.1537172	1.119503	0.1370922	0.3852978
0.65	0.4828277	0.3247191	-0.1536100	1.119266	0.1370389	0.3853736
0.66	0.4827474	0.3246286	-0.1535130	1.119008	0.1369948	0.3854435
0.67	0.4826510	0.3245354	-0.1534267	1.118729	0.1369604	0.3855069
0.68	0.4825379	0.3244392	-0.1533514	1.118427	0.1369361	0.3855637
0.69	0.4824071	0.3243400	-0.1532877	1.118102	0.1369225	0.3856134
0.70	0.4822578	0.3242376	-0.1532362	1.117752	0.1369200	0.3856556
0.71	0.4820891	0.3241318	-0.1531975	1.117376	0.1369292	0.3856899
0.72	0.4818999	0.3240223	-0.1531721	1.116972	0.1369506	0.3857159
0.73	0.4816891	0.3239090	-0.1531608	1.116539	0.1369851	0.3857329
0.74	0.4814557	0.3237916	-0.1531643	1.116076	0.1370331	0.3857405
0.75	0.4811982	0.3236700	-0.1531832	1.115580	0.1370956	0.3857381

Code

summary(sensitivity_moderator)

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.4812   Min.   :0.3237   Min.   :-0.1561  
 1st Qu.:0.5625   1st Qu.:0.4824   1st Qu.:0.3244   1st Qu.:-0.1548  
 Median :0.6250   Median :0.4828   Median :0.3249   Median :-0.1539  
 Mean   :0.6250   Mean   :0.4826   Mean   :0.3249   Mean   :-0.1542  
 3rd Qu.:0.6875   3rd Qu.:0.4829   3rd Qu.:0.3254   3rd Qu.:-0.1533  
 Max.   :0.7500   Max.   :0.4830   Max.   :0.3258   Max.   :-0.1532  
     ci.ub           pvalue           sigma2      
 Min.   :1.116   Min.   :0.1369   Min.   :0.3837  
 1st Qu.:1.118   1st Qu.:0.1370   1st Qu.:0.3845  
 Median :1.120   Median :0.1372   Median :0.3852  
 Mean   :1.119   Mean   :0.1374   Mean   :0.3850  
 3rd Qu.:1.121   3rd Qu.:0.1378   3rd Qu.:0.3856  
 Max.   :1.121   Max.   :0.1386   Max.   :0.3857  

Post measure

Code

##POST MEASURE (second vs. final draft)
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$post_measure <- as.factor(dataN33_prepost$post_measure)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = post_measure)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.2549189	0.3215869	-0.8852175	0.3753798	0.4279581	0.4054589
0.51	-0.2547241	0.3216053	-0.8850588	0.3756106	0.4283376	0.4057681
0.52	-0.2545233	0.3216219	-0.8848907	0.3758441	0.4287257	0.4060730
0.53	-0.2543161	0.3216368	-0.8847127	0.3760804	0.4291228	0.4063734
0.54	-0.2541024	0.3216498	-0.8845244	0.3763195	0.4295294	0.4066692
0.55	-0.2538819	0.3216608	-0.8843255	0.3765616	0.4299457	0.4069602
0.56	-0.2536543	0.3216698	-0.8841155	0.3768068	0.4303721	0.4072461
0.57	-0.2534194	0.3216766	-0.8838940	0.3770552	0.4308091	0.4075267
0.58	-0.2531768	0.3216812	-0.8836604	0.3773068	0.4312570	0.4078018
0.59	-0.2529263	0.3216835	-0.8834144	0.3775619	0.4317163	0.4080711
0.60	-0.2526674	0.3216834	-0.8831553	0.3778205	0.4321876	0.4083344
0.61	-0.2523999	0.3216807	-0.8828826	0.3780827	0.4326713	0.4085913
0.62	-0.2521235	0.3216754	-0.8825956	0.3783487	0.4331679	0.4088415
0.63	-0.2518376	0.3216672	-0.8822938	0.3786186	0.4336780	0.4090847
0.64	-0.2515419	0.3216562	-0.8819764	0.3788927	0.4342023	0.4093206
0.65	-0.2512359	0.3216421	-0.8816428	0.3791709	0.4347413	0.4095488
0.66	-0.2509193	0.3216247	-0.8812921	0.3794536	0.4352957	0.4097688
0.67	-0.2505914	0.3216039	-0.8809235	0.3797408	0.4358664	0.4099803
0.68	-0.2502517	0.3215796	-0.8805361	0.3800327	0.4364539	0.4101828
0.69	-0.2498997	0.3215515	-0.8801291	0.3803296	0.4370593	0.4103757
0.70	-0.2495348	0.3215194	-0.8797013	0.3806317	0.4376833	0.4105586
0.71	-0.2491563	0.3214831	-0.8792517	0.3809390	0.4383270	0.4107308
0.72	-0.2487636	0.3214424	-0.8787791	0.3812520	0.4389912	0.4108918
0.73	-0.2483558	0.3213970	-0.8782823	0.3815707	0.4396772	0.4110409
0.74	-0.2479322	0.3213466	-0.8777599	0.3818955	0.4403861	0.4111773
0.75	-0.2474919	0.3212909	-0.8772104	0.3822266	0.4411191	0.4113003

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :-0.2549   Min.   :0.3213   Min.   :-0.8852  
 1st Qu.:0.5625   1st Qu.:-0.2536   1st Qu.:0.3216   1st Qu.:-0.8841  
 Median :0.6250   Median :-0.2520   Median :0.3216   Median :-0.8824  
 Mean   :0.6250   Mean   :-0.2517   Mean   :0.3216   Mean   :-0.8820  
 3rd Qu.:0.6875   3rd Qu.:-0.2500   3rd Qu.:0.3217   3rd Qu.:-0.8802  
 Max.   :0.7500   Max.   :-0.2475   Max.   :0.3217   Max.   :-0.8772  
     ci.ub            pvalue           sigma2      
 Min.   :0.3754   Min.   :0.4280   Min.   :0.4055  
 1st Qu.:0.3769   1st Qu.:0.4305   1st Qu.:0.4073  
 Median :0.3785   Median :0.4334   Median :0.4090  
 Mean   :0.3786   Mean   :0.4338   Mean   :0.4088  
 3rd Qu.:0.3803   3rd Qu.:0.4369   3rd Qu.:0.4103  
 Max.   :0.3822   Max.   :0.4411   Max.   :0.4113  

Amount

Code

##AMOUNT
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$amount <- as.factor(dataN33_prepost$amount)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = amount)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.0773648	0.3013705	-0.6680401	0.5133104	0.7974026	0.4221423
0.51	-0.0772197	0.3012579	-0.6676743	0.5132348	0.7977003	0.4223859
0.52	-0.0770689	0.3011431	-0.6672985	0.5131607	0.7980116	0.4226248
0.53	-0.0769122	0.3010261	-0.6669124	0.5130880	0.7983368	0.4228588
0.54	-0.0767494	0.3009067	-0.6665158	0.5130169	0.7986764	0.4230877
0.55	-0.0765804	0.3007850	-0.6661082	0.5129474	0.7990307	0.4233113
0.56	-0.0764049	0.3006609	-0.6656894	0.5128796	0.7994003	0.4235294
0.57	-0.0762228	0.3005342	-0.6652591	0.5128134	0.7997857	0.4237416
0.58	-0.0760339	0.3004050	-0.6648169	0.5127490	0.8001872	0.4239478
0.59	-0.0758380	0.3002731	-0.6643625	0.5126864	0.8006056	0.4241477
0.60	-0.0756348	0.3001384	-0.6638953	0.5126257	0.8010412	0.4243410
0.61	-0.0754241	0.3000009	-0.6634152	0.5125669	0.8014948	0.4245273
0.62	-0.0752057	0.2998605	-0.6629215	0.5125101	0.8019670	0.4247064
0.63	-0.0749793	0.2997170	-0.6624139	0.5124553	0.8024583	0.4248779
0.64	-0.0747446	0.2995704	-0.6618918	0.5124027	0.8029695	0.4250414
0.65	-0.0745013	0.2994205	-0.6613548	0.5123522	0.8035013	0.4251966
0.66	-0.0742492	0.2992673	-0.6608023	0.5123040	0.8040546	0.4253430
0.67	-0.0739878	0.2991105	-0.6602337	0.5122581	0.8046301	0.4254801
0.68	-0.0737169	0.2989501	-0.6596483	0.5122146	0.8052287	0.4256075
0.69	-0.0734360	0.2987859	-0.6590456	0.5121736	0.8058515	0.4257246
0.70	-0.0731448	0.2986177	-0.6584248	0.5121352	0.8064992	0.4258309
0.71	-0.0728429	0.2984454	-0.6577852	0.5120994	0.8071732	0.4259257
0.72	-0.0725297	0.2982688	-0.6571259	0.5120664	0.8078744	0.4260085
0.73	-0.0722049	0.2980877	-0.6564461	0.5120363	0.8086043	0.4260785
0.74	-0.0718678	0.2979019	-0.6557448	0.5120092	0.8093639	0.4261349
0.75	-0.0715180	0.2977111	-0.6550211	0.5119851	0.8101549	0.4261769

Code

summary(sensitivity_moderator) 

      ri_t             beta                se             ci.lb        
 Min.   :0.5000   Min.   :-0.07736   Min.   :0.2977   Min.   :-0.6680  
 1st Qu.:0.5625   1st Qu.:-0.07636   1st Qu.:0.2988   1st Qu.:-0.6656  
 Median :0.6250   Median :-0.07509   Median :0.2998   Median :-0.6627  
 Mean   :0.6250   Mean   :-0.07486   Mean   :0.2997   Mean   :-0.6623  
 3rd Qu.:0.6875   3rd Qu.:-0.07351   3rd Qu.:0.3006   3rd Qu.:-0.6592  
 Max.   :0.7500   Max.   :-0.07152   Max.   :0.3014   Max.   :-0.6550  
     ci.ub            pvalue           sigma2      
 Min.   :0.5120   Min.   :0.7974   Min.   :0.4221  
 1st Qu.:0.5122   1st Qu.:0.7995   1st Qu.:0.4236  
 Median :0.5125   Median :0.8022   Median :0.4248  
 Mean   :0.5125   Mean   :0.8028   Mean   :0.4246  
 3rd Qu.:0.5129   3rd Qu.:0.8057   3rd Qu.:0.4257  
 Max.   :0.5133   Max.   :0.8102   Max.   :0.4262  

###Coding

Code

###CODING
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$coding <- as.factor(dataN33_prepost$coding)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = coding)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.2688236	0.2632461	-0.7847764	0.2471291	0.3071655	0.3916269
0.51	-0.2686162	0.2632893	-0.7846537	0.2474214	0.3076185	0.3919788
0.52	-0.2684039	0.2633311	-0.7845233	0.2477156	0.3080778	0.3923265
0.53	-0.2681865	0.2633714	-0.7843849	0.2480119	0.3085437	0.3926697
0.54	-0.2679639	0.2634101	-0.7842382	0.2483103	0.3090164	0.3930084
0.55	-0.2677359	0.2634471	-0.7840828	0.2486110	0.3094963	0.3933423
0.56	-0.2675022	0.2634824	-0.7839183	0.2489139	0.3099836	0.3936712
0.57	-0.2672627	0.2635160	-0.7837445	0.2492192	0.3104787	0.3939948
0.58	-0.2670170	0.2635477	-0.7835609	0.2495269	0.3109818	0.3943130
0.59	-0.2667650	0.2635774	-0.7833671	0.2498372	0.3114933	0.3946255
0.60	-0.2665063	0.2636051	-0.7831627	0.2501501	0.3120136	0.3949319
0.61	-0.2662407	0.2636306	-0.7829472	0.2504659	0.3125430	0.3952321
0.62	-0.2659678	0.2636539	-0.7827200	0.2507844	0.3130820	0.3955257
0.63	-0.2656874	0.2636749	-0.7824808	0.2511059	0.3136309	0.3958125
0.64	-0.2653992	0.2636935	-0.7822288	0.2514305	0.3141903	0.3960919
0.65	-0.2651026	0.2637094	-0.7819636	0.2517584	0.3147606	0.3963638
0.66	-0.2647975	0.2637227	-0.7816845	0.2520895	0.3153423	0.3966277
0.67	-0.2644833	0.2637332	-0.7813908	0.2524242	0.3159359	0.3968831
0.68	-0.2641597	0.2637406	-0.7810818	0.2527624	0.3165421	0.3971296
0.69	-0.2638262	0.2637449	-0.7807567	0.2531044	0.3171615	0.3973668
0.70	-0.2634822	0.2637459	-0.7804147	0.2534503	0.3177947	0.3975941
0.71	-0.2631273	0.2637434	-0.7800549	0.2538003	0.3184424	0.3978109
0.72	-0.2627608	0.2637372	-0.7796763	0.2541546	0.3191054	0.3980167
0.73	-0.2623823	0.2637271	-0.7792778	0.2545133	0.3197845	0.3982107
0.74	-0.2619910	0.2637128	-0.7788585	0.2548766	0.3204806	0.3983923
0.75	-0.2615861	0.2636941	-0.7784170	0.2552448	0.3211946	0.3985606

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :-0.2688   Min.   :0.2632   Min.   :-0.7848  
 1st Qu.:0.5625   1st Qu.:-0.2674   1st Qu.:0.2635   1st Qu.:-0.7839  
 Median :0.6250   Median :-0.2658   Median :0.2637   Median :-0.7826  
 Mean   :0.6250   Mean   :-0.2656   Mean   :0.2636   Mean   :-0.7822  
 3rd Qu.:0.6875   3rd Qu.:-0.2639   3rd Qu.:0.2637   3rd Qu.:-0.7808  
 Max.   :0.7500   Max.   :-0.2616   Max.   :0.2637   Max.   :-0.7784  
     ci.ub            pvalue           sigma2      
 Min.   :0.2471   Min.   :0.3072   Min.   :0.3916  
 1st Qu.:0.2490   1st Qu.:0.3101   1st Qu.:0.3938  
 Median :0.2509   Median :0.3134   Median :0.3957  
 Mean   :0.2510   Mean   :0.3136   Mean   :0.3955  
 3rd Qu.:0.2530   3rd Qu.:0.3170   3rd Qu.:0.3973  
 Max.   :0.2552   Max.   :0.3212   Max.   :0.3986  

###Level of outcome

Code

##LEVEL OF OUTCOME
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$order_outcome <- as.factor(dataN33_prepost$order_outcome)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = order_outcome)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.0402285	0.2089137	-0.4496918	0.3692347	0.8473031	0.4337076
0.51	-0.0402219	0.2089335	-0.4497241	0.3692803	0.8473423	0.4339782
0.52	-0.0402136	0.2089522	-0.4497523	0.3693251	0.8473869	0.4342436
0.53	-0.0402036	0.2089695	-0.4497762	0.3693691	0.8474370	0.4345037
0.54	-0.0401918	0.2089855	-0.4497958	0.3694122	0.8474928	0.4347581
0.55	-0.0401781	0.2090000	-0.4498107	0.3694544	0.8475544	0.4350067
0.56	-0.0401626	0.2090132	-0.4498209	0.3694956	0.8476219	0.4352492
0.57	-0.0401452	0.2090247	-0.4498262	0.3695357	0.8476954	0.4354852
0.58	-0.0401259	0.2090347	-0.4498265	0.3695747	0.8477751	0.4357147
0.59	-0.0401045	0.2090431	-0.4498214	0.3696124	0.8478611	0.4359371
0.60	-0.0400811	0.2090497	-0.4498109	0.3696487	0.8479535	0.4361522
0.61	-0.0400556	0.2090544	-0.4497948	0.3696835	0.8480525	0.4363597
0.62	-0.0400280	0.2090573	-0.4497728	0.3697168	0.8481581	0.4365593
0.63	-0.0399982	0.2090582	-0.4497447	0.3697483	0.8482705	0.4367504
0.64	-0.0399661	0.2090569	-0.4497102	0.3697780	0.8483898	0.4369327
0.65	-0.0399318	0.2090535	-0.4496691	0.3698056	0.8485161	0.4371059
0.66	-0.0398951	0.2090478	-0.4496212	0.3698311	0.8486495	0.4372693
0.67	-0.0398560	0.2090396	-0.4495662	0.3698542	0.8487900	0.4374225
0.68	-0.0398146	0.2090289	-0.4495038	0.3698746	0.8489378	0.4375649
0.69	-0.0397707	0.2090156	-0.4494336	0.3698923	0.8490928	0.4376961
0.70	-0.0397243	0.2089993	-0.4493555	0.3699069	0.8492550	0.4378153
0.71	-0.0396754	0.2089801	-0.4492689	0.3699180	0.8494246	0.4379218
0.72	-0.0396241	0.2089577	-0.4491737	0.3699255	0.8496013	0.4380151
0.73	-0.0395702	0.2089320	-0.4490693	0.3699290	0.8497850	0.4380942
0.74	-0.0395138	0.2089027	-0.4489555	0.3699279	0.8499757	0.4381583
0.75	-0.0394550	0.2088696	-0.4488319	0.3699220	0.8501731	0.4382066

Code

summary(sensitivity_moderator)

      ri_t             beta                se             ci.lb        
 Min.   :0.5000   Min.   :-0.04023   Min.   :0.2089   Min.   :-0.4498  
 1st Qu.:0.5625   1st Qu.:-0.04016   1st Qu.:0.2090   1st Qu.:-0.4498  
 Median :0.6250   Median :-0.04001   Median :0.2090   Median :-0.4497  
 Mean   :0.6250   Mean   :-0.03995   Mean   :0.2090   Mean   :-0.4496  
 3rd Qu.:0.6875   3rd Qu.:-0.03978   3rd Qu.:0.2090   3rd Qu.:-0.4495  
 Max.   :0.7500   Max.   :-0.03945   Max.   :0.2091   Max.   :-0.4488  
     ci.ub            pvalue           sigma2      
 Min.   :0.3692   Min.   :0.8473   Min.   :0.4337  
 1st Qu.:0.3695   1st Qu.:0.8476   1st Qu.:0.4353  
 Median :0.3697   Median :0.8482   Median :0.4367  
 Mean   :0.3697   Mean   :0.8484   Mean   :0.4364  
 3rd Qu.:0.3699   3rd Qu.:0.8491   3rd Qu.:0.4377  
 Max.   :0.3699   Max.   :0.8502   Max.   :0.4382  

###Experiment

Code

##EXPERIMENT
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$experiment <- as.factor(dataN33_prepost$experiment)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = experiment)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.6957096	0.4044346	-1.488387	0.0969677	0.0853956	0.3639792
0.51	-0.6956866	0.4044468	-1.488388	0.0970145	0.0854153	0.3643096
0.52	-0.6956627	0.4044563	-1.488382	0.0970570	0.0854333	0.3646351
0.53	-0.6956381	0.4044630	-1.488371	0.0970947	0.0854496	0.3649554
0.54	-0.6956126	0.4044667	-1.488353	0.0971276	0.0854640	0.3652703
0.55	-0.6955863	0.4044675	-1.488328	0.0971555	0.0854764	0.3655796
0.56	-0.6955591	0.4044652	-1.488296	0.0971780	0.0854868	0.3658829
0.57	-0.6955310	0.4044595	-1.488257	0.0971951	0.0854951	0.3661801
0.58	-0.6955020	0.4044505	-1.488210	0.0972063	0.0855011	0.3664709
0.59	-0.6954720	0.4044378	-1.488156	0.0972116	0.0855049	0.3667549
0.60	-0.6954410	0.4044215	-1.488093	0.0972105	0.0855061	0.3670319
0.61	-0.6954090	0.4044012	-1.488021	0.0972027	0.0855048	0.3673015
0.62	-0.6953760	0.4043769	-1.487940	0.0971880	0.0855009	0.3675634
0.63	-0.6953420	0.4043482	-1.487850	0.0971660	0.0854940	0.3678171
0.64	-0.6953069	0.4043151	-1.487750	0.0971362	0.0854842	0.3680623
0.65	-0.6952707	0.4042773	-1.487640	0.0970982	0.0854712	0.3682985
0.66	-0.6952335	0.4042345	-1.487519	0.0970516	0.0854549	0.3685253
0.67	-0.6951952	0.4041865	-1.487386	0.0969958	0.0854350	0.3687421
0.68	-0.6951559	0.4041330	-1.487242	0.0969303	0.0854113	0.3689484
0.69	-0.6951156	0.4040738	-1.487086	0.0968545	0.0853836	0.3691437
0.70	-0.6950743	0.4040084	-1.486916	0.0967677	0.0853516	0.3693272
0.71	-0.6950321	0.4039366	-1.486733	0.0966690	0.0853149	0.3694984
0.72	-0.6949892	0.4038579	-1.486536	0.0965576	0.0852734	0.3696564
0.73	-0.6949458	0.4037721	-1.486325	0.0964330	0.0852266	0.3698009
0.74	-0.6949019	0.4036784	-1.486097	0.0962933	0.0851738	0.3699300
0.75	-0.6948578	0.4035765	-1.485853	0.0961377	0.0851149	0.3700434

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb       
 Min.   :0.5000   Min.   :-0.6957   Min.   :0.4036   Min.   :-1.488  
 1st Qu.:0.5625   1st Qu.:-0.6956   1st Qu.:0.4041   1st Qu.:-1.488  
 Median :0.6250   Median :-0.6954   Median :0.4044   Median :-1.488  
 Mean   :0.6250   Mean   :-0.6953   Mean   :0.4042   Mean   :-1.488  
 3rd Qu.:0.6875   3rd Qu.:-0.6951   3rd Qu.:0.4044   3rd Qu.:-1.487  
 Max.   :0.7500   Max.   :-0.6949   Max.   :0.4045   Max.   :-1.486  
     ci.ub             pvalue            sigma2      
 Min.   :0.09614   Min.   :0.08511   Min.   :0.3640  
 1st Qu.:0.09687   1st Qu.:0.08539   1st Qu.:0.3660  
 Median :0.09708   Median :0.08545   Median :0.3677  
 Mean   :0.09696   Mean   :0.08541   Mean   :0.3675  
 3rd Qu.:0.09718   3rd Qu.:0.08549   3rd Qu.:0.3691  
 Max.   :0.09721   Max.   :0.08551   Max.   :0.3700  

###System

Code

##SYSTEM
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$system <- as.factor(dataN33_prepost$system_type)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = system)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.1069393	0.1124696	-0.3273757	0.1134971	0.3416914	0.4073082
0.51	-0.1069519	0.1124791	-0.3274068	0.1135031	0.3416754	0.4076081
0.52	-0.1069649	0.1124878	-0.3274370	0.1135071	0.3416539	0.4079024
0.53	-0.1069785	0.1124957	-0.3274661	0.1135091	0.3416268	0.4081909
0.54	-0.1069926	0.1125029	-0.3274941	0.1135090	0.3415937	0.4084734
0.55	-0.1070072	0.1125091	-0.3275210	0.1135066	0.3415546	0.4087497
0.56	-0.1070224	0.1125145	-0.3275467	0.1135020	0.3415092	0.4090194
0.57	-0.1070381	0.1125189	-0.3275712	0.1134949	0.3414572	0.4092822
0.58	-0.1070545	0.1125224	-0.3275944	0.1134854	0.3413983	0.4095380
0.59	-0.1070715	0.1125249	-0.3276162	0.1134732	0.3413324	0.4097863
0.60	-0.1070891	0.1125263	-0.3276366	0.1134584	0.3412590	0.4100268
0.61	-0.1071074	0.1125266	-0.3276555	0.1134407	0.3411779	0.4102592
0.62	-0.1071263	0.1125258	-0.3276728	0.1134201	0.3410888	0.4104830
0.63	-0.1071460	0.1125237	-0.3276885	0.1133964	0.3409913	0.4106980
0.64	-0.1071665	0.1125204	-0.3277024	0.1133695	0.3408851	0.4109036
0.65	-0.1071877	0.1125158	-0.3277145	0.1133392	0.3407696	0.4110993
0.66	-0.1072097	0.1125098	-0.3277248	0.1133054	0.3406447	0.4112848
0.67	-0.1072325	0.1125023	-0.3277329	0.1132679	0.3405096	0.4114594
0.68	-0.1072563	0.1124933	-0.3277390	0.1132265	0.3403641	0.4116227
0.69	-0.1072809	0.1124826	-0.3277428	0.1131810	0.3402075	0.4117739
0.70	-0.1073065	0.1124703	-0.3277442	0.1131312	0.3400393	0.4119125
0.71	-0.1073331	0.1124561	-0.3277431	0.1130769	0.3398589	0.4120378
0.72	-0.1073607	0.1124401	-0.3277393	0.1130178	0.3396657	0.4121490
0.73	-0.1073895	0.1124221	-0.3277327	0.1129537	0.3394589	0.4122452
0.74	-0.1074194	0.1124019	-0.3277231	0.1128843	0.3392377	0.4123256
0.75	-0.1074505	0.1123795	-0.3277103	0.1128092	0.3390014	0.4123893

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :-0.1075   Min.   :0.1124   Min.   :-0.3277  
 1st Qu.:0.5625   1st Qu.:-0.1073   1st Qu.:0.1125   1st Qu.:-0.3277  
 Median :0.6250   Median :-0.1071   Median :0.1125   Median :-0.3277  
 Mean   :0.6250   Mean   :-0.1072   Mean   :0.1125   Mean   :-0.3276  
 3rd Qu.:0.6875   3rd Qu.:-0.1070   3rd Qu.:0.1125   3rd Qu.:-0.3276  
 Max.   :0.7500   Max.   :-0.1069   Max.   :0.1125   Max.   :-0.3274  
     ci.ub            pvalue           sigma2      
 Min.   :0.1128   Min.   :0.3390   Min.   :0.4073  
 1st Qu.:0.1132   1st Qu.:0.3402   1st Qu.:0.4091  
 Median :0.1134   Median :0.3410   Median :0.4106  
 Mean   :0.1133   Mean   :0.3408   Mean   :0.4103  
 3rd Qu.:0.1135   3rd Qu.:0.3415   3rd Qu.:0.4117  
 Max.   :0.1135   Max.   :0.3417   Max.   :0.4124  

###Teacher effects

Code

##teacher_effects
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$teacher <- as.factor(dataN33_prepost$teacher_effects)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = teacher)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.1881239	0.2753030	-0.7277079	0.3514601	0.4943958	0.4219364
0.51	-0.1882207	0.2752857	-0.7277707	0.3513292	0.4941464	0.4222099
0.52	-0.1883190	0.2752665	-0.7278315	0.3511935	0.4938909	0.4224781
0.53	-0.1884189	0.2752456	-0.7278902	0.3510525	0.4936291	0.4227405
0.54	-0.1885202	0.2752227	-0.7279468	0.3509063	0.4933607	0.4229970
0.55	-0.1886232	0.2751978	-0.7280010	0.3507545	0.4930855	0.4232474
0.56	-0.1887279	0.2751708	-0.7280528	0.3505970	0.4928032	0.4234913
0.57	-0.1888343	0.2751417	-0.7281021	0.3504335	0.4925136	0.4237285
0.58	-0.1889425	0.2751103	-0.7281487	0.3502638	0.4922163	0.4239587
0.59	-0.1890525	0.2750765	-0.7281926	0.3500876	0.4919111	0.4241816
0.60	-0.1891645	0.2750403	-0.7282336	0.3499047	0.4915977	0.4243968
0.61	-0.1892784	0.2750016	-0.7282716	0.3497147	0.4912757	0.4246040
0.62	-0.1893944	0.2749601	-0.7283063	0.3495174	0.4909447	0.4248028
0.63	-0.1895126	0.2749158	-0.7283377	0.3493124	0.4906044	0.4249929
0.64	-0.1896330	0.2748686	-0.7283655	0.3490994	0.4902543	0.4251737
0.65	-0.1897558	0.2748182	-0.7283896	0.3488781	0.4898940	0.4253449
0.66	-0.1898809	0.2747646	-0.7284097	0.3486479	0.4895231	0.4255060
0.67	-0.1900086	0.2747077	-0.7284257	0.3484085	0.4891410	0.4256564
0.68	-0.1901389	0.2746471	-0.7284373	0.3481595	0.4887473	0.4257955
0.69	-0.1902720	0.2745827	-0.7284443	0.3479003	0.4883412	0.4259229
0.70	-0.1904079	0.2745144	-0.7284463	0.3476305	0.4879223	0.4260378
0.71	-0.1905469	0.2744419	-0.7284431	0.3473494	0.4874898	0.4261395
0.72	-0.1906890	0.2743650	-0.7284345	0.3470565	0.4870431	0.4262274
0.73	-0.1908344	0.2742834	-0.7284199	0.3467510	0.4865813	0.4263006
0.74	-0.1909834	0.2741968	-0.7283992	0.3464324	0.4861035	0.4263582
0.75	-0.1911360	0.2741049	-0.7283718	0.3460997	0.4856090	0.4263993

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :-0.1911   Min.   :0.2741   Min.   :-0.7284  
 1st Qu.:0.5625   1st Qu.:-0.1902   1st Qu.:0.2746   1st Qu.:-0.7284  
 Median :0.6250   Median :-0.1895   Median :0.2749   Median :-0.7283  
 Mean   :0.6250   Mean   :-0.1895   Mean   :0.2749   Mean   :-0.7282  
 3rd Qu.:0.6875   3rd Qu.:-0.1888   3rd Qu.:0.2752   3rd Qu.:-0.7281  
 Max.   :0.7500   Max.   :-0.1881   Max.   :0.2753   Max.   :-0.7277  
     ci.ub            pvalue           sigma2      
 Min.   :0.3461   Min.   :0.4856   Min.   :0.4219  
 1st Qu.:0.3480   1st Qu.:0.4884   1st Qu.:0.4236  
 Median :0.3494   Median :0.4908   Median :0.4249  
 Mean   :0.3492   Mean   :0.4905   Mean   :0.4246  
 3rd Qu.:0.3506   3rd Qu.:0.4927   3rd Qu.:0.4259  
 Max.   :0.3515   Max.   :0.4944   Max.   :0.4264  

###Validated tool

Code

##validated_tool
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$validated_tool <- as.factor(dataN33_prepost$validated_tool)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = validated_tool)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	1.191959	0.5804574	0.0542833	2.329634	0.0400258	0.3120282
0.51	1.191951	0.5807842	0.0536350	2.330267	0.0401392	0.3124636
0.52	1.191939	0.5811059	0.0529922	2.330885	0.0402518	0.3128936
0.53	1.191922	0.5814222	0.0523550	2.331488	0.0403634	0.3133181
0.54	1.191899	0.5817331	0.0517235	2.332075	0.0404742	0.3137367
0.55	1.191872	0.5820382	0.0510982	2.332646	0.0405839	0.3141492
0.56	1.191839	0.5823374	0.0504790	2.333200	0.0406927	0.3145554
0.57	1.191801	0.5826304	0.0498664	2.333736	0.0408004	0.3149551
0.58	1.191757	0.5829170	0.0492605	2.334253	0.0409071	0.3153478
0.59	1.191707	0.5831972	0.0486610	2.334752	0.0410126	0.3157338
0.60	1.191650	0.5834702	0.0480695	2.335231	0.0411169	0.3161120
0.61	1.191587	0.5837358	0.0474857	2.335688	0.0412198	0.3164823
0.62	1.191517	0.5839938	0.0469099	2.336124	0.0413214	0.3168444
0.63	1.191440	0.5842440	0.0463425	2.336537	0.0414217	0.3171979
0.64	1.191355	0.5844858	0.0457838	2.336926	0.0415204	0.3175425
0.65	1.191262	0.5847190	0.0452343	2.337290	0.0416175	0.3178776
0.66	1.191161	0.5849422	0.0446958	2.337627	0.0417128	0.3182020
0.67	1.191052	0.5851575	0.0441646	2.337940	0.0418067	0.3185178
0.68	1.190934	0.5853620	0.0436455	2.338222	0.0418986	0.3188217
0.69	1.190806	0.5855560	0.0431376	2.338475	0.0419886	0.3191141
0.70	1.190668	0.5857388	0.0426415	2.338696	0.0420764	0.3193945
0.71	1.190520	0.5859100	0.0421579	2.338883	0.0421621	0.3196620
0.72	1.190361	0.5860688	0.0416876	2.339035	0.0422454	0.3199160
0.73	1.190191	0.5862146	0.0412313	2.339150	0.0423262	0.3201556
0.74	1.190008	0.5863466	0.0407900	2.339226	0.0424043	0.3203801
0.75	1.189813	0.5864640	0.0403645	2.339261	0.0424796	0.3205886

Code

summary(sensitivity_moderator)

      ri_t             beta             se             ci.lb        
 Min.   :0.5000   Min.   :1.190   Min.   :0.5805   Min.   :0.04036  
 1st Qu.:0.5625   1st Qu.:1.191   1st Qu.:0.5824   1st Qu.:0.04326  
 Median :0.6250   Median :1.191   Median :0.5841   Median :0.04663  
 Mean   :0.6250   Mean   :1.191   Mean   :0.5839   Mean   :0.04687  
 3rd Qu.:0.6875   3rd Qu.:1.192   3rd Qu.:0.5855   3rd Qu.:0.05033  
 Max.   :0.7500   Max.   :1.192   Max.   :0.5865   Max.   :0.05428  
     ci.ub           pvalue            sigma2      
 Min.   :2.330   Min.   :0.04003   Min.   :0.3120  
 1st Qu.:2.333   1st Qu.:0.04072   1st Qu.:0.3147  
 Median :2.336   Median :0.04137   Median :0.3170  
 Mean   :2.336   Mean   :0.04133   Mean   :0.3168  
 3rd Qu.:2.338   3rd Qu.:0.04197   3rd Qu.:0.3190  
 Max.   :2.339   Max.   :0.04248   Max.   :0.3206  

###Reliability of measurement

Code

##reliability_measurement
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$reliability <- as.factor(dataN33_prepost$reliability_measurement)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = reliability)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.0812447	0.3057183	-0.5179522	0.6804416	0.7904316	0.4349500
0.51	0.0816673	0.3057335	-0.5175593	0.6808939	0.7893774	0.4352120
0.52	0.0820959	0.3057468	-0.5171567	0.6813486	0.7883071	0.4354685
0.53	0.0825310	0.3057581	-0.5167439	0.6818058	0.7872199	0.4357192
0.54	0.0829727	0.3057674	-0.5163203	0.6822658	0.7861149	0.4359641
0.55	0.0834216	0.3057746	-0.5158855	0.6827288	0.7849910	0.4362027
0.56	0.0838781	0.3057795	-0.5154388	0.6831949	0.7838472	0.4364349
0.57	0.0843424	0.3057822	-0.5149796	0.6836645	0.7826823	0.4366604
0.58	0.0848153	0.3057824	-0.5145073	0.6841378	0.7814950	0.4368788
0.59	0.0852970	0.3057802	-0.5140211	0.6846151	0.7802841	0.4370898
0.60	0.0857883	0.3057753	-0.5135202	0.6850968	0.7790480	0.4372932
0.61	0.0862896	0.3057677	-0.5130040	0.6855832	0.7777851	0.4374885
0.62	0.0868017	0.3057571	-0.5124713	0.6860747	0.7764939	0.4376753
0.63	0.0873252	0.3057436	-0.5119214	0.6865717	0.7751725	0.4378533
0.64	0.0878608	0.3057270	-0.5113530	0.6870747	0.7738189	0.4380220
0.65	0.0884095	0.3057070	-0.5107652	0.6875842	0.7724308	0.4381810
0.66	0.0889720	0.3056835	-0.5101567	0.6881007	0.7710060	0.4383297
0.67	0.0895494	0.3056564	-0.5095261	0.6886250	0.7695419	0.4384676
0.68	0.0901428	0.3056254	-0.5088720	0.6891577	0.7680356	0.4385941
0.69	0.0907534	0.3055904	-0.5081928	0.6896996	0.7664840	0.4387086
0.70	0.0913823	0.3055511	-0.5074868	0.6902514	0.7648838	0.4388105
0.71	0.0920312	0.3055072	-0.5067520	0.6908143	0.7632311	0.4388989
0.72	0.0927015	0.3054586	-0.5059864	0.6913893	0.7615219	0.4389733
0.73	0.0933950	0.3054049	-0.5051876	0.6919776	0.7597515	0.4390326
0.74	0.0941137	0.3053458	-0.5043530	0.6925804	0.7579149	0.4390759
0.75	0.0948598	0.3052809	-0.5034799	0.6931995	0.7560063	0.4391024

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :0.08124   Min.   :0.3053   Min.   :-0.5180  
 1st Qu.:0.5625   1st Qu.:0.08399   1st Qu.:0.3056   1st Qu.:-0.5153  
 Median :0.6250   Median :0.08706   Median :0.3057   Median :-0.5122  
 Mean   :0.6250   Mean   :0.08741   Mean   :0.3057   Mean   :-0.5117  
 3rd Qu.:0.6875   3rd Qu.:0.09060   3rd Qu.:0.3058   3rd Qu.:-0.5084  
 Max.   :0.7500   Max.   :0.09486   Max.   :0.3058   Max.   :-0.5035  
     ci.ub            pvalue           sigma2      
 Min.   :0.6804   Min.   :0.7560   Min.   :0.4350  
 1st Qu.:0.6833   1st Qu.:0.7669   1st Qu.:0.4365  
 Median :0.6863   Median :0.7758   Median :0.4378  
 Mean   :0.6865   Mean   :0.7749   Mean   :0.4375  
 3rd Qu.:0.6896   3rd Qu.:0.7836   3rd Qu.:0.4387  
 Max.   :0.6932   Max.   :0.7904   Max.   :0.4391  

Code

#not specified
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$reliable_not <- ifelse(dataN33_prepost$reliability_measurement == 2, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = reliable_not)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue
0.50	0.1223551	0.4390444	-0.7381562	0.9828664	0.7804866
0.51	0.1229361	0.4390222	-0.7375317	0.9834039	0.7794602
0.52	0.1235245	0.4389974	-0.7368946	0.9839437	0.7784200
0.53	0.1241207	0.4389699	-0.7362445	0.9844859	0.7773650
0.54	0.1247251	0.4389396	-0.7355806	0.9850309	0.7762946
0.55	0.1253382	0.4389063	-0.7349023	0.9855787	0.7752079
0.56	0.1259604	0.4388699	-0.7342088	0.9861296	0.7741040
0.57	0.1265923	0.4388304	-0.7334994	0.9866840	0.7729820
0.58	0.1272344	0.4387875	-0.7327733	0.9872420	0.7718406
0.59	0.1278873	0.4387411	-0.7320295	0.9878041	0.7706790
0.60	0.1285517	0.4386911	-0.7312671	0.9883705	0.7694957
0.61	0.1292282	0.4386373	-0.7304852	0.9889416	0.7682896
0.62	0.1299177	0.4385796	-0.7296825	0.9895178	0.7670592
0.63	0.1306209	0.4385176	-0.7288578	0.9900997	0.7658029
0.64	0.1313388	0.4384513	-0.7280100	0.9906877	0.7645192
0.65	0.1320723	0.4383805	-0.7271376	0.9912823	0.7632061
0.66	0.1328225	0.4383048	-0.7262391	0.9918842	0.7618617
0.67	0.1335906	0.4382241	-0.7253128	0.9924940	0.7604839
0.68	0.1343778	0.4381380	-0.7243570	0.9931125	0.7590703
0.69	0.1351855	0.4380464	-0.7233697	0.9937407	0.7576182
0.70	0.1360153	0.4379489	-0.7223487	0.9943793	0.7561249
0.71	0.1368689	0.4378451	-0.7212917	0.9950295	0.7545872
0.72	0.1377481	0.4377347	-0.7201961	0.9956924	0.7530015
0.73	0.1386552	0.4376174	-0.7190590	0.9963695	0.7513641
0.74	0.1395924	0.4374926	-0.7178773	0.9970622	0.7496706
0.75	0.1405624	0.4373600	-0.7166474	0.9977722	0.7479161

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.1224   Min.   :0.4374   Min.   :-0.7382  
 1st Qu.:0.5625   1st Qu.:0.1261   1st Qu.:0.4381   1st Qu.:-0.7340  
 Median :0.6250   Median :0.1303   Median :0.4385   Median :-0.7293  
 Mean   :0.6250   Mean   :0.1307   Mean   :0.4384   Mean   :-0.7286  
 3rd Qu.:0.6875   3rd Qu.:0.1350   3rd Qu.:0.4389   3rd Qu.:-0.7236  
 Max.   :0.7500   Max.   :0.1406   Max.   :0.4390   Max.   :-0.7166  
     ci.ub            pvalue      
 Min.   :0.9829   Min.   :0.7479  
 1st Qu.:0.9863   1st Qu.:0.7580  
 Median :0.9898   Median :0.7664  
 Mean   :0.9900   Mean   :0.7657  
 3rd Qu.:0.9936   3rd Qu.:0.7738  
 Max.   :0.9978   Max.   :0.7805  

Code

#less than .7
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$reliable_less <- ifelse(dataN33_prepost$reliability_measurement == 0, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = reliable_less)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue
0.50	-0.0586480	0.4999269	-1.038487	0.9211907	0.9066120
0.51	-0.0590084	0.5000127	-1.039015	0.9209985	0.9060567
0.52	-0.0593754	0.5000955	-1.039545	0.9207937	0.9054907
0.53	-0.0597492	0.5001750	-1.040074	0.9205758	0.9049136
0.54	-0.0601304	0.5002512	-1.040605	0.9203440	0.9043245
0.55	-0.0605192	0.5003239	-1.041136	0.9200977	0.9037228
0.56	-0.0609161	0.5003930	-1.041668	0.9198361	0.9031077
0.57	-0.0613217	0.5004583	-1.042202	0.9195585	0.9024784
0.58	-0.0617365	0.5005196	-1.042737	0.9192639	0.9018340
0.59	-0.0621611	0.5005768	-1.043274	0.9189514	0.9011736
0.60	-0.0625961	0.5006297	-1.043812	0.9186200	0.9004961
0.61	-0.0630421	0.5006780	-1.044353	0.9182687	0.8998004
0.62	-0.0635000	0.5007216	-1.044896	0.9178963	0.8990853
0.63	-0.0639704	0.5007602	-1.045442	0.9175016	0.8983495
0.64	-0.0644543	0.5007936	-1.045992	0.9170832	0.8975916
0.65	-0.0649526	0.5008216	-1.046545	0.9166397	0.8968101
0.66	-0.0654663	0.5008438	-1.047102	0.9161695	0.8960031
0.67	-0.0659966	0.5008600	-1.047664	0.9156709	0.8951690
0.68	-0.0665446	0.5008698	-1.048231	0.9151421	0.8943055
0.69	-0.0671118	0.5008729	-1.048805	0.9145809	0.8934107
0.70	-0.0676996	0.5008689	-1.049385	0.9139853	0.8924819
0.71	-0.0683096	0.5008574	-1.049972	0.9133528	0.8915166
0.72	-0.0689437	0.5008380	-1.050568	0.9126808	0.8905117
0.73	-0.0696038	0.5008103	-1.051174	0.9119663	0.8894641
0.74	-0.0702921	0.5007736	-1.051790	0.9112062	0.8883700
0.75	-0.0710111	0.5007276	-1.052419	0.9103969	0.8872255

Code

summary(sensitivity_moderator)

      ri_t             beta                se             ci.lb       
 Min.   :0.5000   Min.   :-0.07101   Min.   :0.4999   Min.   :-1.052  
 1st Qu.:0.5625   1st Qu.:-0.06697   1st Qu.:0.5004   1st Qu.:-1.049  
 Median :0.6250   Median :-0.06374   Median :0.5007   Median :-1.045  
 Mean   :0.6250   Mean   :-0.06412   Mean   :0.5006   Mean   :-1.045  
 3rd Qu.:0.6875   3rd Qu.:-0.06102   3rd Qu.:0.5008   3rd Qu.:-1.042  
 Max.   :0.7500   Max.   :-0.05865   Max.   :0.5009   Max.   :-1.038  
     ci.ub            pvalue      
 Min.   :0.9104   Min.   :0.8872  
 1st Qu.:0.9147   1st Qu.:0.8936  
 Median :0.9177   Median :0.8987  
 Mean   :0.9170   Mean   :0.8981  
 3rd Qu.:0.9198   3rd Qu.:0.9030  
 Max.   :0.9212   Max.   :0.9066  

Code

#greater than .7
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$reliable_large <- ifelse(dataN33_prepost$reliability_measurement == 1, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = reliable_large)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue
0.50	-0.0521063	0.3609216	-0.7594997	0.6552871	0.8852082
0.51	-0.0523357	0.3609369	-0.7597590	0.6550875	0.8847111
0.52	-0.0525670	0.3609499	-0.7600159	0.6548818	0.8842093
0.53	-0.0528003	0.3609607	-0.7602703	0.6546697	0.8837027
0.54	-0.0530355	0.3609692	-0.7605221	0.6544511	0.8831909
0.55	-0.0532729	0.3609752	-0.7607713	0.6542256	0.8826739
0.56	-0.0535124	0.3609787	-0.7610178	0.6539929	0.8821513
0.57	-0.0537543	0.3609796	-0.7612613	0.6537527	0.8816229
0.58	-0.0539985	0.3609777	-0.7615018	0.6535048	0.8810884
0.59	-0.0542453	0.3609730	-0.7617393	0.6532487	0.8805475
0.60	-0.0544947	0.3609652	-0.7619735	0.6529841	0.8799998
0.61	-0.0547469	0.3609543	-0.7622043	0.6527105	0.8794452
0.62	-0.0550020	0.3609401	-0.7624316	0.6524276	0.8788831
0.63	-0.0552601	0.3609225	-0.7626552	0.6521350	0.8783131
0.64	-0.0555215	0.3609013	-0.7628750	0.6518320	0.8777349
0.65	-0.0557863	0.3608763	-0.7630908	0.6515182	0.8771480
0.66	-0.0560547	0.3608473	-0.7633025	0.6511930	0.8765517
0.67	-0.0563270	0.3608142	-0.7635099	0.6508559	0.8759457
0.68	-0.0566033	0.3607767	-0.7637128	0.6505061	0.8753292
0.69	-0.0568840	0.3607346	-0.7639109	0.6501429	0.8747016
0.70	-0.0571694	0.3606877	-0.7641043	0.6497656	0.8740620
0.71	-0.0574597	0.3606356	-0.7642925	0.6493732	0.8734098
0.72	-0.0577553	0.3605782	-0.7644756	0.6489649	0.8727438
0.73	-0.0580568	0.3605150	-0.7646532	0.6485396	0.8720631
0.74	-0.0583645	0.3604457	-0.7648251	0.6480962	0.8713665
0.75	-0.0586790	0.3603701	-0.7649913	0.6476334	0.8706525

Code

summary(sensitivity_moderator)

      ri_t             beta                se             ci.lb        
 Min.   :0.5000   Min.   :-0.05868   Min.   :0.3604   Min.   :-0.7650  
 1st Qu.:0.5625   1st Qu.:-0.05681   1st Qu.:0.3607   1st Qu.:-0.7639  
 Median :0.6250   Median :-0.05513   Median :0.3609   Median :-0.7625  
 Mean   :0.6250   Mean   :-0.05522   Mean   :0.3608   Mean   :-0.7624  
 3rd Qu.:0.6875   3rd Qu.:-0.05357   3rd Qu.:0.3610   3rd Qu.:-0.7611  
 Max.   :0.7500   Max.   :-0.05211   Max.   :0.3610   Max.   :-0.7595  
     ci.ub            pvalue      
 Min.   :0.6476   Min.   :0.8707  
 1st Qu.:0.6502   1st Qu.:0.8749  
 Median :0.6523   Median :0.8786  
 Mean   :0.6520   Mean   :0.8784  
 3rd Qu.:0.6539   3rd Qu.:0.8820  
 Max.   :0.6553   Max.   :0.8852  

###Treatment fidelity

Code

##treatment_fidelity
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$fidelity <- as.factor(dataN33_prepost$treatment_fidelity)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = fidelity)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.0007552	0.1971821	-0.3857147	0.3872250	0.9969443	0.4327666
0.51	0.0006399	0.1971904	-0.3858462	0.3871260	0.9974109	0.4330532
0.52	0.0005233	0.1971975	-0.3859766	0.3870233	0.9978825	0.4333346
0.53	0.0004055	0.1972033	-0.3861059	0.3869168	0.9983595	0.4336106
0.54	0.0002862	0.1972079	-0.3862341	0.3868065	0.9988422	0.4338809
0.55	0.0001653	0.1972111	-0.3863613	0.3866919	0.9993311	0.4341453
0.56	0.0000429	0.1972129	-0.3864873	0.3865730	0.9998265	0.4344036
0.57	-0.0000813	0.1972133	-0.3866122	0.3864496	0.9996710	0.4346555
0.58	-0.0002074	0.1972121	-0.3867360	0.3863212	0.9991610	0.4349008
0.59	-0.0003354	0.1972094	-0.3868587	0.3861878	0.9986429	0.4351390
0.60	-0.0004656	0.1972050	-0.3869803	0.3860491	0.9981161	0.4353700
0.61	-0.0005981	0.1971989	-0.3871008	0.3859046	0.9975800	0.4355933
0.62	-0.0007331	0.1971910	-0.3872203	0.3857541	0.9970338	0.4358086
0.63	-0.0008707	0.1971812	-0.3873387	0.3855973	0.9964768	0.4360156
0.64	-0.0010111	0.1971694	-0.3874561	0.3854338	0.9959082	0.4362138
0.65	-0.0011547	0.1971556	-0.3875725	0.3852632	0.9953271	0.4364028
0.66	-0.0013015	0.1971396	-0.3876880	0.3850850	0.9947324	0.4365822
0.67	-0.0014519	0.1971213	-0.3878026	0.3848987	0.9941231	0.4367514
0.68	-0.0016062	0.1971006	-0.3879164	0.3847039	0.9934979	0.4369100
0.69	-0.0017647	0.1970774	-0.3880294	0.3845000	0.9928556	0.4370573
0.70	-0.0019277	0.1970516	-0.3881417	0.3842863	0.9921947	0.4371928
0.71	-0.0020956	0.1970229	-0.3882535	0.3840622	0.9915135	0.4373157
0.72	-0.0022689	0.1969913	-0.3883648	0.3838270	0.9908103	0.4374255
0.73	-0.0024480	0.1969566	-0.3884759	0.3835798	0.9900831	0.4375214
0.74	-0.0026335	0.1969186	-0.3885868	0.3833198	0.9893298	0.4376024
0.75	-0.0028259	0.1968770	-0.3886978	0.3830460	0.9885477	0.4376678

Code

summary(sensitivity_moderator)

      ri_t             beta                  se             ci.lb        
 Min.   :0.5000   Min.   :-2.826e-03   Min.   :0.1969   Min.   :-0.3887  
 1st Qu.:0.5625   1st Qu.:-1.725e-03   1st Qu.:0.1971   1st Qu.:-0.3880  
 Median :0.6250   Median :-8.019e-04   Median :0.1972   Median :-0.3873  
 Mean   :0.6250   Mean   :-8.832e-04   Mean   :0.1971   Mean   :-0.3873  
 3rd Qu.:0.6875   3rd Qu.: 1.183e-05   3rd Qu.:0.1972   3rd Qu.:-0.3865  
 Max.   :0.7500   Max.   : 7.552e-04   Max.   :0.1972   Max.   :-0.3857  
     ci.ub            pvalue           sigma2      
 Min.   :0.3830   Min.   :0.9885   Min.   :0.4328  
 1st Qu.:0.3846   1st Qu.:0.9930   1st Qu.:0.4345  
 Median :0.3857   Median :0.9967   Median :0.4359  
 Mean   :0.3855   Mean   :0.9955   Mean   :0.4357  
 3rd Qu.:0.3865   3rd Qu.:0.9983   3rd Qu.:0.4370  
 Max.   :0.3872   Max.   :0.9998   Max.   :0.4377  

Code

#not specified
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$fidelity_not <- ifelse(dataN33_prepost$treatment_fidelity == 0, 1, 0)
  
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = fidelity_not)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue
0.50	0.1532679	0.3602938	-0.5528951	0.8594308	0.6705474
0.51	0.1534385	0.3603030	-0.5527424	0.8596193	0.6702102
0.52	0.1536067	0.3603100	-0.5525879	0.8598013	0.6698760
0.53	0.1537724	0.3603148	-0.5524316	0.8599765	0.6695451
0.54	0.1539357	0.3603174	-0.5522734	0.8601448	0.6692173
0.55	0.1540963	0.3603176	-0.5521132	0.8603058	0.6688929
0.56	0.1542542	0.3603153	-0.5519508	0.8604593	0.6685717
0.57	0.1544094	0.3603105	-0.5517861	0.8606050	0.6682540
0.58	0.1545618	0.3603029	-0.5516190	0.8607426	0.6679397
0.59	0.1547113	0.3602926	-0.5514492	0.8608718	0.6676289
0.60	0.1548577	0.3602793	-0.5512768	0.8609923	0.6673216
0.61	0.1550011	0.3602630	-0.5511014	0.8611036	0.6670179
0.62	0.1551413	0.3602435	-0.5509229	0.8612056	0.6667179
0.63	0.1552782	0.3602206	-0.5507411	0.8612976	0.6664216
0.64	0.1554118	0.3601942	-0.5505558	0.8613794	0.6661291
0.65	0.1555419	0.3601641	-0.5503668	0.8614505	0.6658403
0.66	0.1556683	0.3601301	-0.5501737	0.8615104	0.6655555
0.67	0.1557911	0.3600921	-0.5499764	0.8615586	0.6652746
0.68	0.1559100	0.3600497	-0.5497745	0.8615946	0.6649976
0.69	0.1560250	0.3600029	-0.5495677	0.8616177	0.6647247
0.70	0.1561359	0.3599513	-0.5493558	0.8616275	0.6644559
0.71	0.1562425	0.3598947	-0.5491382	0.8616231	0.6641912
0.72	0.1563447	0.3598328	-0.5489146	0.8616040	0.6639307
0.73	0.1564424	0.3597652	-0.5486846	0.8615693	0.6636745
0.74	0.1565353	0.3596918	-0.5484477	0.8615182	0.6634225
0.75	0.1566233	0.3596120	-0.5482033	0.8614498	0.6631749

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.1533   Min.   :0.3596   Min.   :-0.5529  
 1st Qu.:0.5625   1st Qu.:0.1543   1st Qu.:0.3600   1st Qu.:-0.5519  
 Median :0.6250   Median :0.1552   Median :0.3602   Median :-0.5508  
 Mean   :0.6250   Mean   :0.1551   Mean   :0.3601   Mean   :-0.5507  
 3rd Qu.:0.6875   3rd Qu.:0.1560   3rd Qu.:0.3603   3rd Qu.:-0.5496  
 Max.   :0.7500   Max.   :0.1566   Max.   :0.3603   Max.   :-0.5482  
     ci.ub            pvalue      
 Min.   :0.8594   Min.   :0.6632  
 1st Qu.:0.8605   1st Qu.:0.6648  
 Median :0.8613   Median :0.6666  
 Mean   :0.8610   Mean   :0.6667  
 3rd Qu.:0.8615   3rd Qu.:0.6685  
 Max.   :0.8616   Max.   :0.6705  

Code

#partially reported
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$fidelity_part <- ifelse(dataN33_prepost$treatment_fidelity == 1, 1, 0)
  
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = fidelity_part)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue
0.50	-0.3013057	0.3493059	-0.9859326	0.3833213	0.3883662
0.51	-0.3012633	0.3493043	-0.9858873	0.3833606	0.3884308
0.52	-0.3012121	0.3493012	-0.9858298	0.3834055	0.3885071
0.53	-0.3011517	0.3492962	-0.9857598	0.3834563	0.3885956
0.54	-0.3010816	0.3492895	-0.9856765	0.3835132	0.3886968
0.55	-0.3010014	0.3492809	-0.9855795	0.3835766	0.3888115
0.56	-0.3009106	0.3492704	-0.9854680	0.3836467	0.3889403
0.57	-0.3008087	0.3492578	-0.9853415	0.3837240	0.3890839
0.58	-0.3006951	0.3492431	-0.9851991	0.3838088	0.3892431
0.59	-0.3005693	0.3492263	-0.9850402	0.3839016	0.3894187
0.60	-0.3004305	0.3492071	-0.9848638	0.3840028	0.3896116
0.61	-0.3002782	0.3491855	-0.9846692	0.3841128	0.3898227
0.62	-0.3001115	0.3491615	-0.9844554	0.3842323	0.3900532
0.63	-0.2999298	0.3491348	-0.9842214	0.3843618	0.3903042
0.64	-0.2997321	0.3491054	-0.9839661	0.3845019	0.3905767
0.65	-0.2995175	0.3490731	-0.9836883	0.3846533	0.3908723
0.66	-0.2992850	0.3490379	-0.9833867	0.3848167	0.3911924
0.67	-0.2990335	0.3489995	-0.9830600	0.3849930	0.3915384
0.68	-0.2987619	0.3489579	-0.9827068	0.3851829	0.3919122
0.69	-0.2984689	0.3489128	-0.9823253	0.3853876	0.3923156
0.70	-0.2981530	0.3488641	-0.9819139	0.3856080	0.3927508
0.71	-0.2978126	0.3488115	-0.9814707	0.3858454	0.3932200
0.72	-0.2974462	0.3487550	-0.9809935	0.3861011	0.3937258
0.73	-0.2970518	0.3486943	-0.9804801	0.3863764	0.3942709
0.74	-0.2966274	0.3486291	-0.9799279	0.3866731	0.3948586
0.75	-0.2961706	0.3485593	-0.9793343	0.3869930	0.3954922

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :-0.3013   Min.   :0.3486   Min.   :-0.9859  
 1st Qu.:0.5625   1st Qu.:-0.3009   1st Qu.:0.3489   1st Qu.:-0.9854  
 Median :0.6250   Median :-0.3000   Median :0.3491   Median :-0.9843  
 Mean   :0.6250   Mean   :-0.2996   Mean   :0.3491   Mean   :-0.9837  
 3rd Qu.:0.6875   3rd Qu.:-0.2985   3rd Qu.:0.3493   3rd Qu.:-0.9824  
 Max.   :0.7500   Max.   :-0.2962   Max.   :0.3493   Max.   :-0.9793  
     ci.ub            pvalue      
 Min.   :0.3833   Min.   :0.3884  
 1st Qu.:0.3837   1st Qu.:0.3890  
 Median :0.3843   Median :0.3902  
 Mean   :0.3846   Mean   :0.3908  
 3rd Qu.:0.3853   3rd Qu.:0.3922  
 Max.   :0.3870   Max.   :0.3955  

Code

#lot of explanations
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$fidelity_yes <- ifelse(dataN33_prepost$treatment_fidelity == 2, 1, 0)
  
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = fidelity_yes)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue
0.50	0.1327394	0.3286621	-0.5114264	0.7769052	0.6863024
0.51	0.1325507	0.3286750	-0.5116404	0.7767417	0.6867363
0.52	0.1323563	0.3286860	-0.5118564	0.7765690	0.6871814
0.53	0.1321559	0.3286951	-0.5120747	0.7763865	0.6876382
0.54	0.1319492	0.3287022	-0.5122952	0.7761937	0.6881073
0.55	0.1317359	0.3287072	-0.5125183	0.7759902	0.6885896
0.56	0.1315157	0.3287100	-0.5127442	0.7757755	0.6890856
0.57	0.1312880	0.3287106	-0.5129730	0.7755490	0.6895963
0.58	0.1310525	0.3287088	-0.5132049	0.7753100	0.6901225
0.59	0.1308088	0.3287046	-0.5134403	0.7750580	0.6906651
0.60	0.1305564	0.3286978	-0.5136795	0.7747923	0.6912252
0.61	0.1302948	0.3286884	-0.5139226	0.7745122	0.6918038
0.62	0.1300234	0.3286761	-0.5141700	0.7742167	0.6924022
0.63	0.1297415	0.3286610	-0.5144222	0.7739052	0.6930216
0.64	0.1294486	0.3286428	-0.5146794	0.7735766	0.6936634
0.65	0.1291439	0.3286214	-0.5149422	0.7732300	0.6943292
0.66	0.1288266	0.3285967	-0.5152110	0.7728643	0.6950206
0.67	0.1284960	0.3285684	-0.5154863	0.7724783	0.6957394
0.68	0.1281510	0.3285366	-0.5157688	0.7720708	0.6964877
0.69	0.1277906	0.3285008	-0.5160591	0.7716404	0.6972676
0.70	0.1274138	0.3284610	-0.5163579	0.7711855	0.6980815
0.71	0.1270193	0.3284169	-0.5166661	0.7707046	0.6989322
0.72	0.1266057	0.3283684	-0.5169845	0.7701959	0.6998225
0.73	0.1261716	0.3283151	-0.5173141	0.7696574	0.7007558
0.74	0.1257153	0.3282568	-0.5176562	0.7690868	0.7017356
0.75	0.1252350	0.3281932	-0.5180119	0.7684818	0.7027661

Code

summary(sensitivity_moderator)

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.1252   Min.   :0.3282   Min.   :-0.5180  
 1st Qu.:0.5625   1st Qu.:0.1279   1st Qu.:0.3285   1st Qu.:-0.5160  
 Median :0.6250   Median :0.1299   Median :0.3287   Median :-0.5143  
 Mean   :0.6250   Mean   :0.1296   Mean   :0.3286   Mean   :-0.5144  
 3rd Qu.:0.6875   3rd Qu.:0.1315   3rd Qu.:0.3287   3rd Qu.:-0.5128  
 Max.   :0.7500   Max.   :0.1327   Max.   :0.3287   Max.   :-0.5114  
     ci.ub            pvalue      
 Min.   :0.7685   Min.   :0.6863  
 1st Qu.:0.7717   1st Qu.:0.6892  
 Median :0.7741   Median :0.6927  
 Mean   :0.7736   Mean   :0.6933  
 3rd Qu.:0.7757   3rd Qu.:0.6971  
 Max.   :0.7769   Max.   :0.7028  

###Writing tasks

Code

##writing_tasks
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$tests <- as.factor(dataN33_prepost$writing_tests)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = tests)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.1864084	0.3376593	-0.8482085	0.4753916	0.5809067	0.4274915
0.51	-0.1865493	0.3376706	-0.8483715	0.4752728	0.5806335	0.4277340
0.52	-0.1866889	0.3376798	-0.8485292	0.4751513	0.5803608	0.4279713
0.53	-0.1868272	0.3376869	-0.8486815	0.4750270	0.5800883	0.4282033
0.54	-0.1869642	0.3376919	-0.8488282	0.4748997	0.5798161	0.4284297
0.55	-0.1871000	0.3376945	-0.8489692	0.4747691	0.5795439	0.4286502
0.56	-0.1872345	0.3376948	-0.8491042	0.4746352	0.5792717	0.4288648
0.57	-0.1873678	0.3376926	-0.8492331	0.4744975	0.5789991	0.4290729
0.58	-0.1874999	0.3376878	-0.8493558	0.4743559	0.5787261	0.4292744
0.59	-0.1876309	0.3376802	-0.8494719	0.4742101	0.5784524	0.4294689
0.60	-0.1877607	0.3376698	-0.8495814	0.4740599	0.5781778	0.4296562
0.61	-0.1878895	0.3376564	-0.8496839	0.4739048	0.5779020	0.4298358
0.62	-0.1880174	0.3376399	-0.8497794	0.4737446	0.5776246	0.4300074
0.63	-0.1881443	0.3376201	-0.8498675	0.4735788	0.5773454	0.4301707
0.64	-0.1882704	0.3375968	-0.8499480	0.4734072	0.5770640	0.4303251
0.65	-0.1883959	0.3375699	-0.8500208	0.4732290	0.5767800	0.4304702
0.66	-0.1885207	0.3375393	-0.8500855	0.4730440	0.5764928	0.4306055
0.67	-0.1886452	0.3375046	-0.8501420	0.4728516	0.5762019	0.4307306
0.68	-0.1887694	0.3374656	-0.8501899	0.4726511	0.5759067	0.4308449
0.69	-0.1888936	0.3374222	-0.8502290	0.4724419	0.5756065	0.4309476
0.70	-0.1890180	0.3373742	-0.8502592	0.4722232	0.5753006	0.4310383
0.71	-0.1891428	0.3373211	-0.8502800	0.4719943	0.5749881	0.4311162
0.72	-0.1892685	0.3372627	-0.8502913	0.4717543	0.5746679	0.4311806
0.73	-0.1893954	0.3371988	-0.8502928	0.4715021	0.5743390	0.4312305
0.74	-0.1895238	0.3371289	-0.8502843	0.4712366	0.5740001	0.4312652
0.75	-0.1896544	0.3370527	-0.8502654	0.4709567	0.5736496	0.4312837

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :-0.1897   Min.   :0.3371   Min.   :-0.8503  
 1st Qu.:0.5625   1st Qu.:-0.1889   1st Qu.:0.3374   1st Qu.:-0.8502  
 Median :0.6250   Median :-0.1881   Median :0.3376   Median :-0.8498  
 Mean   :0.6250   Mean   :-0.1881   Mean   :0.3375   Mean   :-0.8496  
 3rd Qu.:0.6875   3rd Qu.:-0.1873   3rd Qu.:0.3377   3rd Qu.:-0.8491  
 Max.   :0.7500   Max.   :-0.1864   Max.   :0.3377   Max.   :-0.8482  
     ci.ub            pvalue           sigma2      
 Min.   :0.4710   Min.   :0.5736   Min.   :0.4275  
 1st Qu.:0.4725   1st Qu.:0.5757   1st Qu.:0.4289  
 Median :0.4737   Median :0.5775   Median :0.4301  
 Mean   :0.4735   Mean   :0.5774   Mean   :0.4298  
 3rd Qu.:0.4746   3rd Qu.:0.5792   3rd Qu.:0.4309  
 Max.   :0.4754   Max.   :0.5809   Max.   :0.4313  

Code

#self-constructed
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$tests_self <- ifelse(dataN33_prepost$writing_tests == 0, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = tests_self)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue
0.50	0.1864084	0.3376593	-0.4753916	0.8482085	0.5809067
0.51	0.1865493	0.3376706	-0.4752728	0.8483715	0.5806335
0.52	0.1866889	0.3376798	-0.4751513	0.8485292	0.5803608
0.53	0.1868272	0.3376869	-0.4750270	0.8486815	0.5800883
0.54	0.1869642	0.3376919	-0.4748997	0.8488282	0.5798161
0.55	0.1871000	0.3376945	-0.4747691	0.8489692	0.5795439
0.56	0.1872345	0.3376948	-0.4746351	0.8491042	0.5792717
0.57	0.1873678	0.3376926	-0.4744975	0.8492331	0.5789991
0.58	0.1874999	0.3376878	-0.4743559	0.8493558	0.5787261
0.59	0.1876309	0.3376802	-0.4742101	0.8494719	0.5784524
0.60	0.1877607	0.3376698	-0.4740599	0.8495814	0.5781778
0.61	0.1878895	0.3376564	-0.4739048	0.8496839	0.5779020
0.62	0.1880174	0.3376399	-0.4737446	0.8497794	0.5776246
0.63	0.1881443	0.3376201	-0.4735789	0.8498675	0.5773454
0.64	0.1882704	0.3375968	-0.4734071	0.8499480	0.5770640
0.65	0.1883959	0.3375699	-0.4732290	0.8500208	0.5767800
0.66	0.1885207	0.3375393	-0.4730440	0.8500855	0.5764928
0.67	0.1886452	0.3375046	-0.4728516	0.8501420	0.5762019
0.68	0.1887694	0.3374656	-0.4726511	0.8501899	0.5759067
0.69	0.1888936	0.3374223	-0.4724419	0.8502290	0.5756065
0.70	0.1890180	0.3373742	-0.4722232	0.8502592	0.5753006
0.71	0.1891428	0.3373211	-0.4719943	0.8502800	0.5749881
0.72	0.1892685	0.3372627	-0.4717543	0.8502913	0.5746679
0.73	0.1893954	0.3371988	-0.4715021	0.8502928	0.5743390
0.74	0.1895238	0.3371289	-0.4712366	0.8502843	0.5740001
0.75	0.1896544	0.3370526	-0.4709567	0.8502654	0.5736496

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.1864   Min.   :0.3371   Min.   :-0.4754  
 1st Qu.:0.5625   1st Qu.:0.1873   1st Qu.:0.3374   1st Qu.:-0.4746  
 Median :0.6250   Median :0.1881   Median :0.3376   Median :-0.4737  
 Mean   :0.6250   Mean   :0.1881   Mean   :0.3375   Mean   :-0.4735  
 3rd Qu.:0.6875   3rd Qu.:0.1889   3rd Qu.:0.3377   3rd Qu.:-0.4725  
 Max.   :0.7500   Max.   :0.1897   Max.   :0.3377   Max.   :-0.4710  
     ci.ub            pvalue      
 Min.   :0.8482   Min.   :0.5736  
 1st Qu.:0.8491   1st Qu.:0.5757  
 Median :0.8498   Median :0.5775  
 Mean   :0.8496   Mean   :0.5774  
 3rd Qu.:0.8502   3rd Qu.:0.5792  
 Max.   :0.8503   Max.   :0.5809  

Code

#standardized tests
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  data_pp_es <- escalc(measure = "SMCR",
                       m1i=data_pp$m_post, m2i=data_pp$m_pre,
                       sd1i=data_pp$sd_pre, sd2i=data_pp$sd_post,
                       ni=data_pp$ni, ri=rep(ri_t, 33),
                       slab = data_pp$slab)
  # append the id variable
  data_pp_es$id <- data_pp$id
  data_pp_es$tests_norm <- ifelse(dataN33_prepost$writing_tests == 2, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=data_pp_es, random = ~ 1 | id, mods = tests_norm)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue
0.50	-0.1864084	0.3376593	-0.8482085	0.4753916	0.5809067
0.51	-0.1865493	0.3376706	-0.8483715	0.4752728	0.5806335
0.52	-0.1866889	0.3376798	-0.8485292	0.4751513	0.5803608
0.53	-0.1868272	0.3376869	-0.8486815	0.4750270	0.5800883
0.54	-0.1869642	0.3376919	-0.8488282	0.4748997	0.5798161
0.55	-0.1871000	0.3376945	-0.8489692	0.4747691	0.5795439
0.56	-0.1872345	0.3376948	-0.8491042	0.4746351	0.5792717
0.57	-0.1873678	0.3376926	-0.8492331	0.4744975	0.5789991
0.58	-0.1874999	0.3376878	-0.8493558	0.4743559	0.5787261
0.59	-0.1876309	0.3376802	-0.8494719	0.4742102	0.5784524
0.60	-0.1877607	0.3376698	-0.8495814	0.4740599	0.5781778
0.61	-0.1878895	0.3376564	-0.8496839	0.4739048	0.5779020
0.62	-0.1880174	0.3376399	-0.8497794	0.4737446	0.5776246
0.63	-0.1881443	0.3376201	-0.8498675	0.4735788	0.5773454
0.64	-0.1882704	0.3375968	-0.8499480	0.4734071	0.5770640
0.65	-0.1883959	0.3375699	-0.8500208	0.4732290	0.5767800
0.66	-0.1885207	0.3375393	-0.8500855	0.4730440	0.5764928
0.67	-0.1886452	0.3375046	-0.8501420	0.4728516	0.5762019
0.68	-0.1887694	0.3374656	-0.8501899	0.4726511	0.5759067
0.69	-0.1888936	0.3374223	-0.8502290	0.4724419	0.5756065
0.70	-0.1890180	0.3373742	-0.8502592	0.4722232	0.5753006
0.71	-0.1891428	0.3373211	-0.8502800	0.4719943	0.5749881
0.72	-0.1892685	0.3372627	-0.8502913	0.4717543	0.5746679
0.73	-0.1893954	0.3371988	-0.8502928	0.4715021	0.5743390
0.74	-0.1895238	0.3371289	-0.8502843	0.4712366	0.5740001
0.75	-0.1896544	0.3370526	-0.8502654	0.4709567	0.5736496

Code

summary(sensitivity_moderator) 

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :-0.1897   Min.   :0.3371   Min.   :-0.8503  
 1st Qu.:0.5625   1st Qu.:-0.1889   1st Qu.:0.3374   1st Qu.:-0.8502  
 Median :0.6250   Median :-0.1881   Median :0.3376   Median :-0.8498  
 Mean   :0.6250   Mean   :-0.1881   Mean   :0.3375   Mean   :-0.8496  
 3rd Qu.:0.6875   3rd Qu.:-0.1873   3rd Qu.:0.3377   3rd Qu.:-0.8491  
 Max.   :0.7500   Max.   :-0.1864   Max.   :0.3377   Max.   :-0.8482  
     ci.ub            pvalue      
 Min.   :0.4710   Min.   :0.5736  
 1st Qu.:0.4725   1st Qu.:0.5757  
 Median :0.4737   Median :0.5775  
 Mean   :0.4735   Mean   :0.5774  
 3rd Qu.:0.4746   3rd Qu.:0.5792  
 Max.   :0.4754   Max.   :0.5809  

Pre-Post-Control comparison

Code

# prepare data sets for meta-analysis
datT <- data.frame(
  sampleNr = 1:14,
  id = c(1,1,1,2,2,3,3,4,4,5,6,6,7,8),
  m_pre = dataN14_control$M_Feedback,
  m_post = dataN14_control$M_Feedback2,
  sd_pre = dataN14_control$SD_Feedback,
  sd_post = dataN14_control$SD_Feedback2,
  ni = dataN14_control$N_Feedback, 
  slab = dataN14_control$author)

datC <- data.frame(
  sampleNr = 1:14,
  id = c(1,1,1,2,2,3,3,4,4,5,6,6,7,8),
  m_pre = dataN14_control$M_NoFeedback,
  m_post = dataN14_control$M_NoFeedback2,
  sd_pre = dataN14_control$SD_NoFeedback,
  sd_post = dataN14_control$SD_NoFeedback2,
  ni = dataN14_control$N_noFeedback, 
  slab = dataN14_control$author)

Meta-analytic effect

In this section, we used the same approach via sensitivity analyses as above.

Code

sensitivity <- data.frame(ri_t = as.numeric(),  # assumed pre-post correlation treatment group
                          ri_c = as.numeric(),  # assumed pre-post correlation control group
                          pvalue = as.numeric(), # p value of ES
                          beta = as.numeric(),   # meta-analytic ES
                          se = as.numeric(),     # SE of meta-analytic ES
                          sigma2 = as.numeric(),   # tau squared
                          yi.f01 = as.numeric(), # ES of individual studies
                          yi.f02 = as.numeric(),
                          yi.f03 = as.numeric(),
                          yi.f04 = as.numeric(),
                          yi.f05 = as.numeric(),
                          yi.f06 = as.numeric(),
                          yi.f07 = as.numeric(),
                          yi.f08 = as.numeric(),
                          yi.f09 = as.numeric(),
                          yi.f10 = as.numeric(),
                          yi.f11 = as.numeric(),
                          yi.f12 = as.numeric(),
                          yi.f13 = as.numeric(),
                          yi.f14 = as.numeric(),
                          sei.f01 = as.numeric(),  # SEs of ES of individual studies
                          sei.f02 = as.numeric(),
                          sei.f03 = as.numeric(),
                          sei.f04 = as.numeric(),
                          sei.f05 = as.numeric(),
                          sei.f06 = as.numeric(),
                          sei.f07 = as.numeric(),
                          sei.f08 = as.numeric(),
                          sei.f09 = as.numeric(),
                          sei.f10 = as.numeric(),
                          sei.f11 = as.numeric(),
                          sei.f12 = as.numeric(),
                          sei.f13 = as.numeric(),
                          sei.f14 = as.numeric()
                          )

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from between-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  rma_overall_clustered <- rma.mv(yi, vi, data=dat, random = ~ 1 | id)
  
  sensitivity <- sensitivity %>%
    add_row(ri_t = ri_t, 
            ri_c = (ri_t+.14),
            pvalue = rma_overall_clustered$pval,
            beta = rma_overall_clustered$beta[,1],
            se = rma_overall_clustered$se[1],
            sigma2 = rma_overall_clustered$sigma2[1],
            yi.f01 = rma_overall_clustered$yi.f[1],
            yi.f02 = rma_overall_clustered$yi.f[2],
            yi.f03 = rma_overall_clustered$yi.f[3],
            yi.f04 = rma_overall_clustered$yi.f[4],
            yi.f05 = rma_overall_clustered$yi.f[5],
            yi.f06 = rma_overall_clustered$yi.f[6],
            yi.f07 = rma_overall_clustered$yi.f[7],
            yi.f08 = rma_overall_clustered$yi.f[8],
            yi.f09 = rma_overall_clustered$yi.f[9],
            yi.f10 = rma_overall_clustered$yi.f[10],
            yi.f11 = rma_overall_clustered$yi.f[11],
            yi.f12 = rma_overall_clustered$yi.f[12],
            yi.f13 = rma_overall_clustered$yi.f[13],
            yi.f14 = rma_overall_clustered$yi.f[14],
            sei.f01 = sqrt(dat$vi[1]),
            sei.f02 = sqrt(dat$vi[2]),
            sei.f03 = sqrt(dat$vi[3]),
            sei.f04 = sqrt(dat$vi[4]),
            sei.f05 = sqrt(dat$vi[5]),
            sei.f06 = sqrt(dat$vi[6]),
            sei.f07 = sqrt(dat$vi[7]),
            sei.f08 = sqrt(dat$vi[8]),
            sei.f09 = sqrt(dat$vi[9]),
            sei.f10 = sqrt(dat$vi[10]),
            sei.f11 = sqrt(dat$vi[11]),
            sei.f12 = sqrt(dat$vi[12]),
            sei.f13 = sqrt(dat$vi[13]),
            sei.f14 = sqrt(dat$vi[14])
    )
}

Overview of meta-analytic ES

Results from all 26 meta-analyses:

• ri_t: assumed pre-post-correlation (treatment group)
• ri_c: assumed pre-post-correlation (control group)
• beta: meta-analytic ES
• se: SE of meta-analytic ES
• pvalue: p value of meta-analytic ES

Code

sensitivity %>%
  dplyr::select(ri_t, ri_c, beta, se, pvalue, sigma2) %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	ri_c	beta	se	pvalue	sigma2
0.50	0.64	0.1224893	0.2240719	0.5846181	0.3377329
0.51	0.65	0.1224397	0.2237719	0.5842668	0.3374673
0.52	0.66	0.1223842	0.2234692	0.5839282	0.3371949
0.53	0.67	0.1223224	0.2231634	0.5836030	0.3369149
0.54	0.68	0.1222540	0.2228544	0.5832920	0.3366269
0.55	0.69	0.1221786	0.2225421	0.5829962	0.3363302
0.56	0.70	0.1220958	0.2222263	0.5827166	0.3360245
0.57	0.71	0.1220050	0.2219068	0.5824544	0.3357091
0.58	0.72	0.1219059	0.2215833	0.5822107	0.3353834
0.59	0.73	0.1217979	0.2212558	0.5819869	0.3350467
0.60	0.74	0.1216805	0.2209239	0.5817844	0.3346983
0.61	0.75	0.1215529	0.2205873	0.5816046	0.3343374
0.62	0.76	0.1214147	0.2202458	0.5814493	0.3339631
0.63	0.77	0.1212650	0.2198991	0.5813203	0.3335743
0.64	0.78	0.1211030	0.2195468	0.5812193	0.3331701
0.65	0.79	0.1209280	0.2191885	0.5811486	0.3327492
0.66	0.80	0.1207390	0.2188238	0.5811105	0.3323103
0.67	0.81	0.1205349	0.2184522	0.5811073	0.3318520
0.68	0.82	0.1203148	0.2180733	0.5811420	0.3313726
0.69	0.83	0.1200774	0.2176863	0.5812173	0.3308704
0.70	0.84	0.1198214	0.2172907	0.5813366	0.3303433
0.71	0.85	0.1195452	0.2168857	0.5815034	0.3297891
0.72	0.86	0.1192475	0.2164705	0.5817216	0.3292052
0.73	0.87	0.1189263	0.2160441	0.5819955	0.3285887
0.74	0.88	0.1185798	0.2156056	0.5823298	0.3279364
0.75	0.89	0.1182058	0.2151536	0.5827295	0.3272446

We summarize these results:

• ES_mean: meta-analytic ES (mean over the 26 meta-analyses)
• ES_min: meta-analytic ES (min of the 26 meta-analyses)
• ES_max: meta-analytic ES (max of the 26 meta-analyses)
• CI_lower_mean: lower CI of meta-analytic ES (mean over the 26 meta-analyses)
• CI_lower_min: lower CI of meta-analytic ES (min of the 26 meta-analyses)
• CI_lower_max: lower CI of meta-analytic ES (max of the 26 meta-analyses)
• CI_upper_mean: upper CI of meta-analytic ES (mean over the 26 meta-analyses)
• CI_upper_min: upper CI of meta-analytic ES (min of the 26 meta-analyses)
• CI_upper_max: upper CI of meta-analytic ES (max of the 26 meta-analyses)
• pvalue_mean: p value of meta-analytic ES (mean over the 26 meta-analyses)
• pvalue_min: p value of meta-analytic ES (min of the 26 meta-analyses)
• pvalue_max: p value of meta-analytic ES (max of the 26 meta-analyses)

Code

# compute mean, min and max of ES, CI and pvalue from meta-analysis
sensitivity %>%
  dplyr::summarise(ES_mean = mean(beta),
                   ES_min  = min(beta),
                   ES_max  = max(beta),
                   CI_lower_mean        = mean(beta-(1.96*se)),
                   CI_lower_min = min(beta-(1.96*se)),
                   CI_lower_max = max(beta-(1.96*se)),
                   CI_upper_mean        = mean(beta+(1.96*se)),
                   CI_upper_min = min(beta+(1.96*se)),
                   CI_upper_max = max(beta+(1.96*se)),
                   pvalue_mean        = mean(pvalue),
                   pvalue_min = min(pvalue),
                   pvalue_max = max(pvalue),
                   sigma2_mean = mean(sigma2)) %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover")) %>%
  column_spec(1, background = "#C74A48") %>%
  column_spec(2, background = "#B54341") %>%
  column_spec(3, background = "#913634") %>%
  column_spec(4, background = "#27A357") %>%
  column_spec(5, background = "#208748") %>%
  column_spec(6, background = "#186636") %>%
  column_spec(7, background = "#B244B8") %>%
  column_spec(8, background = "#8A358F") %>%
  column_spec(9, background = "#6A296E") %>%
  column_spec(10, background = "#9E8A47") %>%
  column_spec(11, background = "#8F7C40") %>%
  column_spec(12, background = "#6E6031")

ES_mean	ES_min	ES_max	CI_lower_mean	CI_lower_min	CI_lower_max	CI_upper_mean	CI_upper_min	CI_upper_max	pvalue_mean	pvalue_min	pvalue_max	sigma2_mean
0.1209927	0.1182058	0.1224893	-0.3100356	-0.3166917	-0.3034952	0.5520209	0.5399069	0.5616703	0.5821843	0.5811073	0.5846181	0.3333245

We summarize the same aspects for all 22 studies:
Note that the ES does not vary due to different pre-post-correlation, but SE does.

Code

# compute mean, min and max of ES and CI from each study
sensitivity %>%
  dplyr::select(-c(beta, pvalue, se, sigma2)) %>%
  pivot_longer(c(3:30),                       # reshape data data from
               names_to = "variable",         # sensitivity analysis
               values_to = "values") %>%
  mutate(sampleNr = as.numeric(str_sub(variable, -2, -1)),
         variable = str_sub(variable, 1, -5)) %>%
  pivot_wider(id_cols = c(sampleNr, ri_t, ri_c), 
              names_from = "variable", 
              values_from = "values") %>%
  group_by(sampleNr) %>%
  dplyr::summarise(ES = mean(yi),    # compute mean, min and max
                   CI_lower_mean = mean(yi-(1.96*sei)),
                   CI_lower_min  = min(yi-(1.96*sei)),
                   CI_lower_max  = max(yi-(1.96*sei)),
                   CI_upper_mean = mean(yi+(1.96*sei)),
                   CI_upper_min  = min(yi+(1.96*sei)),
                   CI_upper_max  = max(yi+(1.96*sei)))  %>%
  right_join(datT[c("sampleNr", "slab")],., by = "sampleNr") %>%
  dplyr::select(-sampleNr) %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover")) %>%
  column_spec(2, background = "#C74A48") %>%
  column_spec(3, background = "#27A357") %>%
  column_spec(4, background = "#208748") %>%
  column_spec(5, background = "#186636") %>%
  column_spec(6, background = "#B244B8") %>%
  column_spec(7, background = "#8A358F") %>%
  column_spec(8, background = "#6A296E")

slab	ES	CI_lower_mean	CI_lower_min	CI_lower_max	CI_upper_mean	CI_upper_min	CI_upper_max
Burkhart et al., 2020	0.6814068	0.0528333	-0.0054387	0.1147960	1.3099802	1.2480176	1.3682523
Burkhart et al., 2020	-0.0702724	-0.6009093	-0.6721562	-0.5229026	0.4603646	0.3823579	0.5316114
Burkhart et al., 2020	-0.1818085	-0.6830836	-0.7537560	-0.6053296	0.3194665	0.2417125	0.3901389
Frost, 2008	-0.0504075	-0.6613129	-0.7785211	-0.5273539	0.5604978	0.4265388	0.6777060
Frost, 2008	0.6438224	0.0885290	-0.0212781	0.2146052	1.1991159	1.0730397	1.3089229
Kellogg et al., 2010	-1.0744334	-2.6117714	-2.6436196	-2.5794951	0.4629046	0.4306283	0.4947528
Kellogg et al., 2010	-0.8138863	-2.3964347	-2.4273843	-2.3650927	0.7686620	0.7373200	0.7996117
Lachner et al., 2017b (Study 2)	0.8725517	0.2120163	0.1448799	0.2838472	1.5330871	1.4612562	1.6002236
Lachner et al., 2017b (Study 3)	0.4871669	-0.1650553	-0.2692787	-0.0487959	1.1393891	1.0231297	1.2436125
Lachner & Neuburg, 2019	0.3769218	-0.1191822	-0.1803689	-0.0527118	0.8730258	0.8065553	0.9342125
Palermo, 2017	0.6407101	0.5039751	0.4793736	0.5318357	0.7774452	0.7495846	0.8020467
Palermo, 2017	1.1696352	1.0175403	0.9957871	1.0415086	1.3217301	1.2977617	1.3434833
Roscoe et al., 2013	-0.0231180	-0.4029033	-0.4765134	-0.3186452	0.3566673	0.2724092	0.4302774
Niloy et al., 2023	-0.7949552	-0.9304498	-0.9529573	-0.9052246	-0.6594606	-0.6846858	-0.6369531

Forest Plot

In order to be able to display a forst plot, we calculated a meta-analysis with the mean assumed pre-post-correlation (r=.625 for the intervention group; r=.765 for the control group).

Code

## FOREST PLOT with mean correlation
###compute effect sizes for writing quality
# effect sizes from within-subjects data
datT_es<- escalc(measure = "SMCR",
                 m1i=datT$m_post, m2i=datT$m_pre,
                 sd1i=datT$sd_pre, sd2i=datT$sd_post,
                 ni=datT$ni, ri=rep(.625, 14),
                 slab = datT$slab)

datT_es$id <- datT$id

datC_es <- escalc(measure="SMCR",
                  m1i=datC$m_post, m2i=datC$m_pre,
                  sd1i=datC$sd_pre, sd2i=datC$sd_post,
                  ni=datC$ni, ri=rep((.625+.14), 14),
                  slab = datC$slab)
datC_es$id<-datC$id

dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)

dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)

REM <- rma.mv(yi, vi, data=dat, random = ~ 1 | id)

###FOREST PLOT
forest(REM)

Bias estimation

Funnel plot

Code

##Funnel plot
funnel(REM, legend = T)

Trim and fill

Code

##Trim & fill
rma_trimfill <- rma(yi, vi, data=dat) # compute random-effects model without
                                      # clustering, as trimfill() can't handle
                                      # these objects
taf_overall <- trimfill(rma_trimfill)
taf_overall 

Estimated number of missing studies on the right side: 1 (SE = 2.5068)

Random-Effects Model (k = 15; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0.3226 (SE = 0.1585)
tau (square root of estimated tau^2 value):      0.5679
I^2 (total heterogeneity / total variability):   92.54%
H^2 (total variability / sampling variability):  13.40

Test for Heterogeneity:
Q(df = 14) = 418.6696, p-val < .0001

Model Results:

estimate      se    zval    pval    ci.lb   ci.ub    
  0.2687  0.1701  1.5796  0.1142  -0.0647  0.6022    

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Code

funnel(taf_overall, legend = T)

Egger’s test

Code

##Egger's test
regtest(rma_trimfill) 

Regression Test for Funnel Plot Asymmetry

Model:     mixed-effects meta-regression model
Predictor: standard error

Test for Funnel Plot Asymmetry: z = -1.2397, p = 0.2151
Limit Estimate (as sei -> 0):   b = 0.5502 (CI: -0.0663, 1.1667)

Heterogeneity

Calculating \(I^2\)

Code

###HETEROGENEITY

# Establish empty data frame to be filled with results
heterogeneity_sen <- data.frame(ri_t = as.numeric(),   # assumed pre-post correlation (treatment group)
                                ri_c = as.numeric(),   # assumed pre-post correlation (control group)
                                I2 = as.numeric())     # I²

# starting loop over 26 possible pre-post-correlations
for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from between-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  REM <- rma.mv(yi, vi, 
                data=dat, 
                random = ~ 1 | id # take clustered data into account
                )
  
  # Formula
  W <- diag(1/REM$vi)
  X <- model.matrix(REM)
  P <- W-W%*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W
  I2 <- 100*REM$sigma2/(sum(REM$sigma2)+(REM$k - REM$p)/sum(diag(P)))
  
  # save estimates for sensitivity analysis
  heterogeneity_sen <- heterogeneity_sen %>%
    add_row(ri_t = ri_t, 
            ri_c = ri_t+.14,
            I2 = I2)
}

heterogeneity_sen %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	ri_c	I2
0.50	0.64	91.28634
0.51	0.65	91.43733
0.52	0.66	91.58902
0.53	0.67	91.74141
0.54	0.68	91.89453
0.55	0.69	92.04837
0.56	0.70	92.20295
0.57	0.71	92.35829
0.58	0.72	92.51440
0.59	0.73	92.67129
0.60	0.74	92.82897
0.61	0.75	92.98747
0.62	0.76	93.14681
0.63	0.77	93.30699
0.64	0.78	93.46806
0.65	0.79	93.63001
0.66	0.80	93.79289
0.67	0.81	93.95672
0.68	0.82	94.12153
0.69	0.83	94.28736
0.70	0.84	94.45424
0.71	0.85	94.62221
0.72	0.86	94.79133
0.73	0.87	94.96165
0.74	0.88	95.13322
0.75	0.89	95.30613

Code

skim(heterogeneity_sen)

Data summary

Name	heterogeneity_sen
Number of rows	26
Number of columns	3
_______________________
Column type frequency:
numeric	3
________________________
Group variables	None

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
ri_t	0	1	0.625	0.076	0.500	0.562	0.625	0.688	0.750	▇▇▇▇▇
ri_c	0	1	0.765	0.076	0.640	0.703	0.765	0.827	0.890	▇▇▇▇▇
I2	0	1	93.252	1.228	91.286	92.242	93.227	94.246	95.306	▇▇▇▇▇

Moderators

We will report results from all 26 meta-analyses for the moderators.

For each moderator we provide

• a table with the main parameters from each meta-analysis
  • ri_t: assumed pre-post-correlation (sensitivity analysis)
  • beta: ES of moderator
  • ci.lb: lower bound CI of ES of moderator
  • ci.ub: upper bound CI of ES of moderator
  • pvalue: p value of ES of moderator
• a summary table of these parameters

Representation

Graphical representation

Code

#graphical representation
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$rep_g <- as.factor(dataN14_control$rep_graphical)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = rep_g)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.4645432	0.2991202	-0.1217217	1.0508081	0.1204156	0.3155087
0.51	0.4636580	0.2972652	-0.1189711	1.0462871	0.1188201	0.3149844
0.52	0.4627455	0.2953844	-0.1161973	1.0416883	0.1172112	0.3144511
0.53	0.4618045	0.2934769	-0.1133998	1.0370087	0.1155885	0.3139082
0.54	0.4608336	0.2915420	-0.1105783	1.0322455	0.1139520	0.3133559
0.55	0.4598314	0.2895787	-0.1077324	1.0273952	0.1123016	0.3127934
0.56	0.4587964	0.2875859	-0.1048616	1.0224545	0.1106369	0.3122202
0.57	0.4577270	0.2855627	-0.1019657	1.0174196	0.1089579	0.3116357
0.58	0.4566213	0.2835080	-0.0990441	1.0122867	0.1072644	0.3110394
0.59	0.4554775	0.2814205	-0.0960966	1.0070516	0.1055561	0.3104306
0.60	0.4542937	0.2792992	-0.0931227	1.0017101	0.1038331	0.3098085
0.61	0.4530677	0.2771427	-0.0901221	0.9962574	0.1020950	0.3091725
0.62	0.4517971	0.2749497	-0.0870944	0.9906886	0.1003417	0.3085216
0.63	0.4504796	0.2727187	-0.0840391	0.9849984	0.0985731	0.3078551
0.64	0.4491125	0.2704481	-0.0809561	0.9791811	0.0967891	0.3071718
0.65	0.4476929	0.2681365	-0.0778449	0.9732307	0.0949895	0.3064707
0.66	0.4462178	0.2657820	-0.0747053	0.9671408	0.0931742	0.3057506
0.67	0.4446837	0.2633827	-0.0715369	0.9609044	0.0913430	0.3050101
0.68	0.4430872	0.2609368	-0.0683396	0.9545141	0.0894958	0.3042478
0.69	0.4414244	0.2584422	-0.0651130	0.9479617	0.0876327	0.3034619
0.70	0.4396909	0.2558965	-0.0618570	0.9412388	0.0857533	0.3026508
0.71	0.4378822	0.2532973	-0.0585715	0.9343359	0.0838578	0.3018122
0.72	0.4359933	0.2506422	-0.0552563	0.9272429	0.0819460	0.3009440
0.73	0.4340187	0.2479282	-0.0519116	0.9199490	0.0800179	0.3000434
0.74	0.4319524	0.2451523	-0.0485372	0.9124421	0.0780734	0.2991076
0.75	0.4297879	0.2423113	-0.0451335	0.9047094	0.0761125	0.2981333

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb         
 Min.   :0.5000   Min.   :0.4298   Min.   :0.2423   Min.   :-0.12172  
 1st Qu.:0.5625   1st Qu.:0.4418   1st Qu.:0.2591   1st Qu.:-0.10414  
 Median :0.6250   Median :0.4511   Median :0.2738   Median :-0.08557  
 Mean   :0.6250   Mean   :0.4497   Mean   :0.2727   Mean   :-0.08480  
 3rd Qu.:0.6875   3rd Qu.:0.4585   3rd Qu.:0.2871   3rd Qu.:-0.06592  
 Max.   :0.7500   Max.   :0.4645   Max.   :0.2991   Max.   :-0.04513  
     ci.ub            pvalue            sigma2      
 Min.   :0.9047   Min.   :0.07611   Min.   :0.2981  
 1st Qu.:0.9496   1st Qu.:0.08810   1st Qu.:0.3037  
 Median :0.9878   Median :0.09946   Median :0.3082  
 Mean   :0.9843   Mean   :0.09903   Mean   :0.3077  
 3rd Qu.:1.0212   3rd Qu.:0.11022   3rd Qu.:0.3121  
 Max.   :1.0508   Max.   :0.12042   Max.   :0.3155  

Numerical representation

Code

#numeric representation
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$rep_n <- as.factor(dataN14_control$rep_numeric)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = rep_n)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.1396910	0.4739078	-0.7891512	1.068533	0.7681741	0.3831438
0.51	0.1391449	0.4734058	-0.7887135	1.067003	0.7688168	0.3830100
0.52	0.1386065	0.4728972	-0.7882550	1.065468	0.7694452	0.3828662
0.53	0.1380763	0.4723817	-0.7877748	1.063927	0.7700586	0.3827121
0.54	0.1375548	0.4718589	-0.7872717	1.062381	0.7706562	0.3825470
0.55	0.1370422	0.4713284	-0.7867446	1.060829	0.7712370	0.3823702
0.56	0.1365390	0.4707899	-0.7861922	1.059270	0.7718001	0.3821811
0.57	0.1360458	0.4702428	-0.7856132	1.057705	0.7723443	0.3819790
0.58	0.1355631	0.4696868	-0.7850061	1.056132	0.7728687	0.3817629
0.59	0.1350914	0.4691212	-0.7843693	1.054552	0.7733721	0.3815322
0.60	0.1346313	0.4685455	-0.7837011	1.052964	0.7738530	0.3812856
0.61	0.1341834	0.4679591	-0.7829997	1.051366	0.7743101	0.3810223
0.62	0.1337484	0.4673613	-0.7822630	1.049760	0.7747420	0.3807411
0.63	0.1333270	0.4667514	-0.7814889	1.048143	0.7751471	0.3804407
0.64	0.1329201	0.4661285	-0.7806750	1.046515	0.7755234	0.3801198
0.65	0.1325283	0.4654917	-0.7798186	1.044875	0.7758693	0.3797768
0.66	0.1321527	0.4648400	-0.7789168	1.043222	0.7761826	0.3794100
0.67	0.1317943	0.4641723	-0.7779667	1.041555	0.7764610	0.3790175
0.68	0.1314540	0.4634874	-0.7769646	1.039872	0.7767022	0.3785974
0.69	0.1311330	0.4627838	-0.7759067	1.038173	0.7769033	0.3781473
0.70	0.1308325	0.4620602	-0.7747888	1.036454	0.7770615	0.3776646
0.71	0.1305540	0.4613147	-0.7736061	1.034714	0.7771735	0.3771465
0.72	0.1302990	0.4605454	-0.7723534	1.032951	0.7772357	0.3765897
0.73	0.1300690	0.4597501	-0.7710247	1.031163	0.7772440	0.3759906
0.74	0.1298659	0.4589264	-0.7696133	1.029345	0.7771940	0.3753451
0.75	0.1296919	0.4580715	-0.7681117	1.027495	0.7770804	0.3746485

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.1297   Min.   :0.4581   Min.   :-0.7892  
 1st Qu.:0.5625   1st Qu.:0.1312   1st Qu.:0.4630   1st Qu.:-0.7860  
 Median :0.6250   Median :0.1335   Median :0.4671   Median :-0.7819  
 Mean   :0.6250   Mean   :0.1339   Mean   :0.4667   Mean   :-0.7807  
 3rd Qu.:0.6875   3rd Qu.:0.1364   3rd Qu.:0.4707   3rd Qu.:-0.7762  
 Max.   :0.7500   Max.   :0.1397   Max.   :0.4739   Max.   :-0.7681  
     ci.ub           pvalue           sigma2      
 Min.   :1.027   Min.   :0.7682   Min.   :0.3746  
 1st Qu.:1.039   1st Qu.:0.7719   1st Qu.:0.3783  
 Median :1.049   Median :0.7749   Median :0.3806  
 Mean   :1.049   Mean   :0.7741   Mean   :0.3800  
 3rd Qu.:1.059   3rd Qu.:0.7769   3rd Qu.:0.3821  
 Max.   :1.069   Max.   :0.7772   Max.   :0.3831  

Highlighting representation

Code

#highlighting representation
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$rep_h <- as.factor(dataN14_control$rep_highlighting)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = rep_h)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.2321299	0.3242949	-0.4034764	0.8677362	0.4741155	0.3501718
0.51	0.2313462	0.3221528	-0.4000617	0.8627540	0.4726798	0.3497470
0.52	0.2305327	0.3199823	-0.3966212	0.8576865	0.4712453	0.3493126
0.53	0.2296878	0.3177827	-0.3931549	0.8525305	0.4698135	0.3488678
0.54	0.2288100	0.3155529	-0.3896624	0.8472824	0.4683857	0.3484124
0.55	0.2278974	0.3132921	-0.3861437	0.8419386	0.4669638	0.3479456
0.56	0.2269483	0.3109991	-0.3825987	0.8364953	0.4655495	0.3474669
0.57	0.2259605	0.3086729	-0.3790273	0.8309483	0.4641449	0.3469756
0.58	0.2249321	0.3063124	-0.3754293	0.8252934	0.4627523	0.3464710
0.59	0.2238605	0.3039165	-0.3718049	0.8195259	0.4613744	0.3459523
0.60	0.2227435	0.3014838	-0.3681539	0.8136410	0.4600138	0.3454188
0.61	0.2215784	0.2990131	-0.3644766	0.8076333	0.4586738	0.3448695
0.62	0.2203622	0.2965030	-0.3607729	0.8014974	0.4573581	0.3443034
0.63	0.2190920	0.2939519	-0.3570430	0.7952271	0.4560704	0.3437190
0.64	0.2177645	0.2913583	-0.3532874	0.7888163	0.4548154	0.3431160
0.65	0.2163760	0.2887207	-0.3495062	0.7822581	0.4535980	0.3424928
0.66	0.2149227	0.2860371	-0.3456998	0.7755452	0.4524238	0.3418478
0.67	0.2134004	0.2833059	-0.3418689	0.7686697	0.4512991	0.3411795
0.68	0.2118047	0.2805248	-0.3380139	0.7616232	0.4502310	0.3404863
0.69	0.2101305	0.2776919	-0.3341357	0.7543967	0.4492275	0.3397661
0.70	0.2083725	0.2748049	-0.3302351	0.7469802	0.4482976	0.3390168
0.71	0.2065250	0.2718612	-0.3263133	0.7393632	0.4474515	0.3382362
0.72	0.2045814	0.2688584	-0.3223713	0.7315342	0.4467011	0.3374214
0.73	0.2025349	0.2657935	-0.3184108	0.7234806	0.4460593	0.3365697
0.74	0.2003779	0.2626636	-0.3144333	0.7151890	0.4455416	0.3356775
0.75	0.1981019	0.2594653	-0.3104408	0.7066446	0.4451651	0.3347412

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.1981   Min.   :0.2595   Min.   :-0.4035  
 1st Qu.:0.5625   1st Qu.:0.2105   1st Qu.:0.2784   1st Qu.:-0.3817  
 Median :0.6250   Median :0.2197   Median :0.2952   Median :-0.3589  
 Mean   :0.6250   Mean   :0.2181   Mean   :0.2940   Mean   :-0.3582  
 3rd Qu.:0.6875   3rd Qu.:0.2267   3rd Qu.:0.3104   3rd Qu.:-0.3351  
 Max.   :0.7500   Max.   :0.2321   Max.   :0.3243   Max.   :-0.3104  
     ci.ub            pvalue           sigma2      
 Min.   :0.7066   Min.   :0.4452   Min.   :0.3347  
 1st Qu.:0.7562   1st Qu.:0.4495   1st Qu.:0.3399  
 Median :0.7984   Median :0.4567   Median :0.3440  
 Mean   :0.7944   Mean   :0.4577   Mean   :0.3435  
 3rd Qu.:0.8351   3rd Qu.:0.4652   3rd Qu.:0.3473  
 Max.   :0.8677   Max.   :0.4741   Max.   :0.3502  

Text-based representation

Code

#text-based representation
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$rep_t <- as.factor(dataN14_control$rep_text_based)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = rep_t)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.1592562	0.4844770	-1.108814	0.7903013	0.7423690	0.3909857
0.51	-0.1594807	0.4838907	-1.107889	0.7889277	0.7417173	0.3906129
0.52	-0.1597059	0.4832982	-1.106953	0.7875411	0.7410599	0.3902311
0.53	-0.1599319	0.4826993	-1.106005	0.7861413	0.7403966	0.3898403
0.54	-0.1601588	0.4820931	-1.105044	0.7847262	0.7397265	0.3894389
0.55	-0.1603869	0.4814795	-1.104069	0.7832955	0.7390493	0.3890268
0.56	-0.1606163	0.4808581	-1.103081	0.7818483	0.7383644	0.3886032
0.57	-0.1608472	0.4802286	-1.102078	0.7803835	0.7376712	0.3881676
0.58	-0.1610799	0.4795903	-1.101060	0.7788997	0.7369690	0.3877191
0.59	-0.1613147	0.4789427	-1.100025	0.7773958	0.7362570	0.3872568
0.60	-0.1615519	0.4782854	-1.098974	0.7758702	0.7355342	0.3867798
0.61	-0.1617920	0.4776176	-1.097905	0.7743212	0.7347995	0.3862871
0.62	-0.1620354	0.4769386	-1.096818	0.7727470	0.7340519	0.3857775
0.63	-0.1622826	0.4762476	-1.095711	0.7711456	0.7332899	0.3852498
0.64	-0.1625343	0.4755439	-1.094583	0.7695145	0.7325118	0.3847025
0.65	-0.1627912	0.4748264	-1.093434	0.7678515	0.7317161	0.3841342
0.66	-0.1630542	0.4740941	-1.092262	0.7661532	0.7309005	0.3835431
0.67	-0.1633243	0.4733458	-1.091065	0.7644165	0.7300626	0.3829273
0.68	-0.1636028	0.4725803	-1.089843	0.7626376	0.7291996	0.3822847
0.69	-0.1638910	0.4717960	-1.088594	0.7608121	0.7283083	0.3816129
0.70	-0.1641906	0.4709912	-1.087316	0.7589353	0.7273848	0.3809092
0.71	-0.1645036	0.4701642	-1.086008	0.7570014	0.7264248	0.3801706
0.72	-0.1648324	0.4693128	-1.084669	0.7550038	0.7254228	0.3793937
0.73	-0.1651798	0.4684345	-1.083295	0.7529350	0.7243726	0.3785745
0.74	-0.1655494	0.4675267	-1.081885	0.7507860	0.7232668	0.3777087
0.75	-0.1659453	0.4665861	-1.080437	0.7485465	0.7220963	0.3767909

Code

summary(sensitivity_moderator) 

      ri_t             beta               se             ci.lb       
 Min.   :0.5000   Min.   :-0.1659   Min.   :0.4666   Min.   :-1.109  
 1st Qu.:0.5625   1st Qu.:-0.1638   1st Qu.:0.4720   1st Qu.:-1.103  
 Median :0.6250   Median :-0.1622   Median :0.4766   Median :-1.096  
 Mean   :0.6250   Mean   :-0.1623   Mean   :0.4762   Mean   :-1.096  
 3rd Qu.:0.6875   3rd Qu.:-0.1607   3rd Qu.:0.4807   3rd Qu.:-1.089  
 Max.   :0.7500   Max.   :-0.1593   Max.   :0.4845   Max.   :-1.080  
     ci.ub            pvalue           sigma2      
 Min.   :0.7485   Min.   :0.7221   Min.   :0.3768  
 1st Qu.:0.7613   1st Qu.:0.7285   1st Qu.:0.3818  
 Median :0.7719   Median :0.7337   Median :0.3855  
 Mean   :0.7711   Mean   :0.7332   Mean   :0.3850  
 3rd Qu.:0.7815   3rd Qu.:0.7382   3rd Qu.:0.3885  
 Max.   :0.7903   Max.   :0.7424   Max.   :0.3910  

Multiple representation formats

Code

#number of representations
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$rep_nr <- as.factor(dataN14_control$rep_nr)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = rep_nr)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.3724921	0.2900898	-0.1960734	0.9410576	0.1991216	0.2900436
0.51	0.3700428	0.2885467	-0.1954984	0.9355840	0.1996897	0.2902134
0.52	0.3675557	0.2869769	-0.1949086	0.9300201	0.2002691	0.2903765
0.53	0.3650292	0.2853794	-0.1943040	0.9243625	0.2008612	0.2905325
0.54	0.3624613	0.2837532	-0.1936848	0.9186074	0.2014673	0.2906811
0.55	0.3598499	0.2820975	-0.1930511	0.9127510	0.2020890	0.2908221
0.56	0.3571929	0.2804113	-0.1924031	0.9067890	0.2027280	0.2909550
0.57	0.3544879	0.2786934	-0.1917412	0.9007170	0.2033860	0.2910794
0.58	0.3517324	0.2769428	-0.1910655	0.8945303	0.2040653	0.2911950
0.59	0.3489237	0.2751583	-0.1903766	0.8882240	0.2047681	0.2913012
0.60	0.3460590	0.2733386	-0.1896749	0.8817928	0.2054970	0.2913977
0.61	0.3431351	0.2714825	-0.1889608	0.8752309	0.2062548	0.2914838
0.62	0.3401487	0.2695885	-0.1882350	0.8685324	0.2070448	0.2915591
0.63	0.3370963	0.2676551	-0.1874981	0.8616907	0.2078705	0.2916228
0.64	0.3339740	0.2656809	-0.1867510	0.8546989	0.2087358	0.2916744
0.65	0.3307775	0.2636641	-0.1859946	0.8475496	0.2096453	0.2917131
0.66	0.3275025	0.2616030	-0.1852299	0.8402349	0.2106039	0.2917381
0.67	0.3241439	0.2594957	-0.1844582	0.8327461	0.2116173	0.2917485
0.68	0.3206966	0.2573401	-0.1836808	0.8250740	0.2126920	0.2917433
0.69	0.3171546	0.2551343	-0.1828994	0.8172086	0.2138350	0.2917216
0.70	0.3135118	0.2528758	-0.1821156	0.8091391	0.2150546	0.2916819
0.71	0.3097611	0.2505621	-0.1813316	0.8008539	0.2163602	0.2916232
0.72	0.3058952	0.2481908	-0.1805498	0.7923401	0.2177623	0.2915438
0.73	0.3019056	0.2457588	-0.1797728	0.7835840	0.2192732	0.2914421
0.74	0.2977833	0.2432632	-0.1790037	0.7745704	0.2209069	0.2913163
0.75	0.2935184	0.2407006	-0.1782461	0.7652828	0.2226796	0.2911643

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.2935   Min.   :0.2407   Min.   :-0.1961  
 1st Qu.:0.5625   1st Qu.:0.3180   1st Qu.:0.2557   1st Qu.:-0.1922  
 Median :0.6250   Median :0.3386   Median :0.2686   Median :-0.1879  
 Mean   :0.6250   Mean   :0.3366   Mean   :0.2675   Mean   :-0.1876  
 3rd Qu.:0.6875   3rd Qu.:0.3565   3rd Qu.:0.2800   3rd Qu.:-0.1831  
 Max.   :0.7500   Max.   :0.3725   Max.   :0.2901   Max.   :-0.1782  
     ci.ub            pvalue           sigma2      
 Min.   :0.7653   Min.   :0.1991   Min.   :0.2900  
 1st Qu.:0.8192   1st Qu.:0.2029   1st Qu.:0.2910  
 Median :0.8651   Median :0.2075   Median :0.2914  
 Mean   :0.8609   Mean   :0.2086   Mean   :0.2912  
 3rd Qu.:0.9053   3rd Qu.:0.2135   3rd Qu.:0.2917  
 Max.   :0.9411   Max.   :0.2227   Max.   :0.2917  

Level of feedback

Code

## LEVEL ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$FB_order <- as.factor(dataN14_control$FB_order)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = FB_order)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.5200935	0.1157803	-0.7470188	-0.2931682	7.1e-06	0.3071398
0.51	-0.5201700	0.1147119	-0.7450012	-0.2953388	5.8e-06	0.3071772
0.52	-0.5202486	0.1136304	-0.7429601	-0.2975371	4.7e-06	0.3072155
0.53	-0.5203294	0.1125355	-0.7408948	-0.2997639	3.8e-06	0.3072548
0.54	-0.5204124	0.1114266	-0.7388045	-0.3020203	3.0e-06	0.3072952
0.55	-0.5204977	0.1103034	-0.7366884	-0.3043070	2.4e-06	0.3073367
0.56	-0.5205852	0.1091654	-0.7345455	-0.3066250	1.9e-06	0.3073791
0.57	-0.5206752	0.1080120	-0.7323749	-0.3089755	1.4e-06	0.3074230
0.58	-0.5207675	0.1068428	-0.7301755	-0.3113594	1.1e-06	0.3074682
0.59	-0.5208622	0.1056572	-0.7279464	-0.3137779	8.0e-07	0.3075148
0.60	-0.5209594	0.1044545	-0.7256864	-0.3162323	6.0e-07	0.3075629
0.61	-0.5210590	0.1032341	-0.7233942	-0.3187239	4.0e-07	0.3076126
0.62	-0.5211613	0.1019954	-0.7210686	-0.3212539	3.0e-07	0.3076639
0.63	-0.5212661	0.1007377	-0.7187083	-0.3238239	2.0e-07	0.3077170
0.64	-0.5213736	0.0994601	-0.7163117	-0.3264354	2.0e-07	0.3077721
0.65	-0.5214837	0.0981618	-0.7138774	-0.3290900	1.0e-07	0.3078293
0.66	-0.5215966	0.0968421	-0.7114036	-0.3317896	1.0e-07	0.3078888
0.67	-0.5217123	0.0954998	-0.7088885	-0.3345361	0.0e+00	0.3079508
0.68	-0.5218308	0.0941341	-0.7063303	-0.3373314	0.0e+00	0.3080156
0.69	-0.5219523	0.0927438	-0.7037267	-0.3401778	0.0e+00	0.3080835
0.70	-0.5220767	0.0913277	-0.7010757	-0.3430776	0.0e+00	0.3081548
0.71	-0.5222041	0.0898847	-0.6983748	-0.3460334	0.0e+00	0.3082299
0.72	-0.5223346	0.0884132	-0.6956212	-0.3490479	0.0e+00	0.3083094
0.73	-0.5224682	0.0869118	-0.6928122	-0.3521242	0.0e+00	0.3083939
0.74	-0.5226050	0.0853789	-0.6899445	-0.3552656	0.0e+00	0.3084841
0.75	-0.5227452	0.0838126	-0.6870148	-0.3584755	0.0e+00	0.3085811

Code

summary(sensitivity_moderator) 

      ri_t             beta               se              ci.lb        
 Min.   :0.5000   Min.   :-0.5227   Min.   :0.08381   Min.   :-0.7470  
 1st Qu.:0.5625   1st Qu.:-0.5219   1st Qu.:0.09309   1st Qu.:-0.7340  
 Median :0.6250   Median :-0.5212   Median :0.10137   Median :-0.7199  
 Mean   :0.6250   Mean   :-0.5213   Mean   :0.10081   Mean   :-0.7189  
 3rd Qu.:0.6875   3rd Qu.:-0.5206   3rd Qu.:0.10888   3rd Qu.:-0.7044  
 Max.   :0.7500   Max.   :-0.5201   Max.   :0.11578   Max.   :-0.6870  
     ci.ub             pvalue              sigma2      
 Min.   :-0.3585   Min.   :4.460e-10   Min.   :0.3071  
 1st Qu.:-0.3395   1st Qu.:2.109e-08   1st Qu.:0.3074  
 Median :-0.3225   Median :2.756e-07   Median :0.3077  
 Mean   :-0.3237   Mean   :1.305e-06   Mean   :0.3077  
 3rd Qu.:-0.3072   3rd Qu.:1.748e-06   3rd Qu.:0.3081  
 Max.   :-0.2932   Max.   :7.053e-06   Max.   :0.3086  

Higher level only

Code

## HIGHER LEVEL ONLY ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$order_hi <- as.factor(ifelse(dataN14_control$FB_order == 2, 1, 0))
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = order_hi)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.4963895	0.1179525	-0.7275722	-0.2652069	2.57e-05	0.3901608
0.51	-0.4969280	0.1168266	-0.7259040	-0.2679520	2.10e-05	0.3904365
0.52	-0.4974700	0.1156879	-0.7242141	-0.2707260	1.71e-05	0.3907100
0.53	-0.4980156	0.1145358	-0.7225016	-0.2735295	1.37e-05	0.3909811
0.54	-0.4985646	0.1133700	-0.7207657	-0.2763635	1.09e-05	0.3912493
0.55	-0.4991172	0.1121900	-0.7190056	-0.2792288	8.60e-06	0.3915143
0.56	-0.4996734	0.1109954	-0.7172204	-0.2821263	6.70e-06	0.3917759
0.57	-0.5002331	0.1097857	-0.7154092	-0.2850571	5.20e-06	0.3920335
0.58	-0.5007964	0.1085603	-0.7135708	-0.2880221	4.00e-06	0.3922868
0.59	-0.5013633	0.1073188	-0.7117042	-0.2910224	3.00e-06	0.3925352
0.60	-0.5019338	0.1060604	-0.7098084	-0.2940592	2.20e-06	0.3927784
0.61	-0.5025078	0.1047847	-0.7078820	-0.2971337	1.60e-06	0.3930157
0.62	-0.5030855	0.1034908	-0.7059238	-0.3002472	1.20e-06	0.3932465
0.63	-0.5036668	0.1021782	-0.7039325	-0.3034011	8.00e-07	0.3934702
0.64	-0.5042517	0.1008461	-0.7019065	-0.3065969	6.00e-07	0.3936861
0.65	-0.5048402	0.0994937	-0.6998443	-0.3098361	4.00e-07	0.3938935
0.66	-0.5054323	0.0981201	-0.6977442	-0.3131205	3.00e-07	0.3940914
0.67	-0.5060281	0.0967244	-0.6956044	-0.3164519	2.00e-07	0.3942790
0.68	-0.5066275	0.0953056	-0.6934230	-0.3198321	1.00e-07	0.3944551
0.69	-0.5072306	0.0938626	-0.6911979	-0.3232633	1.00e-07	0.3946188
0.70	-0.5078373	0.0923943	-0.6889268	-0.3267478	0.00e+00	0.3947688
0.71	-0.5084477	0.0908995	-0.6866074	-0.3302880	0.00e+00	0.3949036
0.72	-0.5090617	0.0893767	-0.6842369	-0.3338865	0.00e+00	0.3950217
0.73	-0.5096794	0.0878246	-0.6818124	-0.3375463	0.00e+00	0.3951219
0.74	-0.5103007	0.0862415	-0.6793309	-0.3412705	0.00e+00	0.3952014
0.75	-0.5109257	0.0846256	-0.6767889	-0.3450625	0.00e+00	0.3952585

Code

summary(sensitivity_moderator) 

      ri_t             beta               se              ci.lb        
 Min.   :0.5000   Min.   :-0.5109   Min.   :0.08463   Min.   :-0.7276  
 1st Qu.:0.5625   1st Qu.:-0.5071   1st Qu.:0.09422   1st Qu.:-0.7168  
 Median :0.6250   Median :-0.5034   Median :0.10283   Median :-0.7049  
 Mean   :0.6250   Mean   :-0.5035   Mean   :0.10229   Mean   :-0.7040  
 3rd Qu.:0.6875   3rd Qu.:-0.4998   3rd Qu.:0.11069   3rd Qu.:-0.6918  
 Max.   :0.7500   Max.   :-0.4964   Max.   :0.11795   Max.   :-0.6768  
     ci.ub             pvalue              sigma2      
 Min.   :-0.3451   Min.   :1.565e-09   Min.   :0.3902  
 1st Qu.:-0.3224   1st Qu.:7.543e-08   1st Qu.:0.3918  
 Median :-0.3018   Median :9.961e-07   Median :0.3934  
 Mean   :-0.3030   Mean   :4.750e-06   Mean   :0.3931  
 3rd Qu.:-0.2829   3rd Qu.:6.355e-06   3rd Qu.:0.3946  
 Max.   :-0.2652   Max.   :2.572e-05   Max.   :0.3953  

Both lower and higher level

Code

## BOTH LOWER AND HIGHER LEVEL ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$order_hilow <- ifelse(dataN14_control$FB_order == 3, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = order_hilow)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.3428719	0.5027374	-1.328219	0.6424754	0.4952327	0.3837353
0.51	-0.3422861	0.5020673	-1.326320	0.6417477	0.4953949	0.3835347
0.52	-0.3416840	0.5013923	-1.324395	0.6410269	0.4955739	0.3833278
0.53	-0.3410648	0.5007128	-1.322444	0.6403143	0.4957712	0.3831151
0.54	-0.3404276	0.5000283	-1.320465	0.6396099	0.4959876	0.3828958
0.55	-0.3397712	0.4993385	-1.318457	0.6389143	0.4962243	0.3826694
0.56	-0.3390946	0.4986432	-1.316417	0.6382281	0.4964826	0.3824356
0.57	-0.3383965	0.4979420	-1.314345	0.6375518	0.4967639	0.3821938
0.58	-0.3376757	0.4972345	-1.312237	0.6368861	0.4970698	0.3819435
0.59	-0.3369306	0.4965205	-1.310093	0.6362317	0.4974018	0.3816841
0.60	-0.3361598	0.4957996	-1.307909	0.6355895	0.4977618	0.3814149
0.61	-0.3353615	0.4950712	-1.305683	0.6349602	0.4981518	0.3811352
0.62	-0.3345338	0.4943349	-1.303413	0.6343448	0.4985740	0.3808442
0.63	-0.3336747	0.4935903	-1.301094	0.6337445	0.4990308	0.3805412
0.64	-0.3327819	0.4928366	-1.298724	0.6331602	0.4995248	0.3802251
0.65	-0.3318528	0.4920733	-1.296299	0.6325932	0.5000589	0.3798949
0.66	-0.3308847	0.4912997	-1.293815	0.6320451	0.5006366	0.3795495
0.67	-0.3298745	0.4905150	-1.291266	0.6315172	0.5012613	0.3791875
0.68	-0.3288188	0.4897183	-1.288649	0.6310115	0.5019372	0.3788077
0.69	-0.3277136	0.4889086	-1.285957	0.6305298	0.5026690	0.3784083
0.70	-0.3265545	0.4880849	-1.283183	0.6300744	0.5034619	0.3779876
0.71	-0.3253367	0.4872459	-1.280321	0.6296478	0.5043217	0.3775436
0.72	-0.3240545	0.4863903	-1.277362	0.6292530	0.5052554	0.3770741
0.73	-0.3227014	0.4855165	-1.274296	0.6288934	0.5062706	0.3765765
0.74	-0.3212702	0.4846227	-1.271113	0.6285729	0.5073764	0.3760481
0.75	-0.3197524	0.4837074	-1.267801	0.6282966	0.5085834	0.3754861

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb       
 Min.   :0.5000   Min.   :-0.3429   Min.   :0.4837   Min.   :-1.328  
 1st Qu.:0.5625   1st Qu.:-0.3389   1st Qu.:0.4891   1st Qu.:-1.316  
 Median :0.6250   Median :-0.3341   Median :0.4940   Median :-1.302  
 Mean   :0.6250   Mean   :-0.3331   Mean   :0.4937   Mean   :-1.301  
 3rd Qu.:0.6875   3rd Qu.:-0.3280   3rd Qu.:0.4985   3rd Qu.:-1.287  
 Max.   :0.7500   Max.   :-0.3198   Max.   :0.5027   Max.   :-1.268  
     ci.ub            pvalue           sigma2      
 Min.   :0.6283   Min.   :0.4952   Min.   :0.3755  
 1st Qu.:0.6307   1st Qu.:0.4966   1st Qu.:0.3785  
 Median :0.6340   Median :0.4988   Median :0.3807  
 Mean   :0.6345   Mean   :0.4999   Mean   :0.3803  
 3rd Qu.:0.6381   3rd Qu.:0.5025   3rd Qu.:0.3824  
 Max.   :0.6425   Max.   :0.5086   Max.   :0.3837  

Specificity

Code

## SPECIFICITY ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$spec <- as.factor(dataN14_control$FB_specificity)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = spec)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.3428719	0.5027374	-0.6424754	1.328219	0.4952327	0.3837353
0.51	0.3422861	0.5020673	-0.6417477	1.326320	0.4953949	0.3835347
0.52	0.3416840	0.5013923	-0.6410269	1.324395	0.4955739	0.3833278
0.53	0.3410648	0.5007128	-0.6403143	1.322444	0.4957712	0.3831151
0.54	0.3404276	0.5000283	-0.6396099	1.320465	0.4959876	0.3828958
0.55	0.3397712	0.4993385	-0.6389143	1.318457	0.4962243	0.3826694
0.56	0.3390946	0.4986432	-0.6382281	1.316417	0.4964826	0.3824356
0.57	0.3383965	0.4979420	-0.6375518	1.314345	0.4967639	0.3821938
0.58	0.3376757	0.4972345	-0.6368861	1.312237	0.4970698	0.3819435
0.59	0.3369306	0.4965205	-0.6362317	1.310093	0.4974018	0.3816841
0.60	0.3361598	0.4957996	-0.6355895	1.307909	0.4977618	0.3814149
0.61	0.3353615	0.4950712	-0.6349602	1.305683	0.4981518	0.3811352
0.62	0.3345338	0.4943349	-0.6343448	1.303413	0.4985740	0.3808442
0.63	0.3336747	0.4935903	-0.6337445	1.301094	0.4990308	0.3805412
0.64	0.3327819	0.4928366	-0.6331602	1.298724	0.4995248	0.3802251
0.65	0.3318528	0.4920733	-0.6325932	1.296299	0.5000589	0.3798949
0.66	0.3308847	0.4912997	-0.6320451	1.293815	0.5006366	0.3795495
0.67	0.3298745	0.4905150	-0.6315172	1.291266	0.5012613	0.3791875
0.68	0.3288188	0.4897183	-0.6310115	1.288649	0.5019372	0.3788077
0.69	0.3277136	0.4889087	-0.6305298	1.285957	0.5026690	0.3784083
0.70	0.3265545	0.4880849	-0.6300744	1.283183	0.5034619	0.3779876
0.71	0.3253367	0.4872459	-0.6296478	1.280321	0.5043217	0.3775436
0.72	0.3240545	0.4863903	-0.6292530	1.277362	0.5052554	0.3770741
0.73	0.3227014	0.4855165	-0.6288934	1.274296	0.5062706	0.3765765
0.74	0.3212702	0.4846227	-0.6285729	1.271113	0.5073764	0.3760481
0.75	0.3197524	0.4837073	-0.6282966	1.267801	0.5085834	0.3754861

Code

summary(sensitivity_moderator)

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.3198   Min.   :0.4837   Min.   :-0.6425  
 1st Qu.:0.5625   1st Qu.:0.3280   1st Qu.:0.4891   1st Qu.:-0.6381  
 Median :0.6250   Median :0.3341   Median :0.4940   Median :-0.6340  
 Mean   :0.6250   Mean   :0.3331   Mean   :0.4937   Mean   :-0.6345  
 3rd Qu.:0.6875   3rd Qu.:0.3389   3rd Qu.:0.4985   3rd Qu.:-0.6307  
 Max.   :0.7500   Max.   :0.3429   Max.   :0.5027   Max.   :-0.6283  
     ci.ub           pvalue           sigma2      
 Min.   :1.268   Min.   :0.4952   Min.   :0.3755  
 1st Qu.:1.287   1st Qu.:0.4966   1st Qu.:0.3785  
 Median :1.302   Median :0.4988   Median :0.3807  
 Mean   :1.301   Mean   :0.4999   Mean   :0.3803  
 3rd Qu.:1.316   3rd Qu.:0.5025   3rd Qu.:0.3824  
 Max.   :1.328   Max.   :0.5086   Max.   :0.3837  

Tool numbers

Code

## TOOL NUMBERS ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$tool_nr <- as.factor(dataN14_control$FB_tool_numbers)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = tool_nr)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.1335501	0.1545890	-0.4365390	0.1694389	0.3876407	0.3506367
0.51	-0.1335603	0.1544060	-0.4361905	0.1690700	0.3870421	0.3501995
0.52	-0.1335693	0.1542209	-0.4358366	0.1686981	0.3864404	0.3497538
0.53	-0.1335771	0.1540335	-0.4354773	0.1683231	0.3858353	0.3492990
0.54	-0.1335836	0.1538439	-0.4351121	0.1679448	0.3852266	0.3488348
0.55	-0.1335889	0.1536518	-0.4347409	0.1675630	0.3846139	0.3483604
0.56	-0.1335929	0.1534571	-0.4343633	0.1671774	0.3839969	0.3478754
0.57	-0.1335955	0.1532596	-0.4339789	0.1667878	0.3833752	0.3473789
0.58	-0.1335968	0.1530593	-0.4335875	0.1663939	0.3827485	0.3468704
0.59	-0.1335966	0.1528559	-0.4331886	0.1659954	0.3821162	0.3463489
0.60	-0.1335950	0.1526491	-0.4327818	0.1655918	0.3814777	0.3458136
0.61	-0.1335919	0.1524389	-0.4323666	0.1651828	0.3808324	0.3452635
0.62	-0.1335873	0.1522249	-0.4319426	0.1647680	0.3801795	0.3446975
0.63	-0.1335812	0.1520069	-0.4315092	0.1643467	0.3795182	0.3441145
0.64	-0.1335737	0.1517845	-0.4310659	0.1639185	0.3788474	0.3435132
0.65	-0.1335647	0.1515575	-0.4306119	0.1634826	0.3781661	0.3428920
0.66	-0.1335542	0.1513255	-0.4301467	0.1630383	0.3774727	0.3422493
0.67	-0.1335423	0.1510881	-0.4296695	0.1625848	0.3767659	0.3415834
0.68	-0.1335292	0.1508447	-0.4291794	0.1621210	0.3760436	0.3408922
0.69	-0.1335148	0.1505950	-0.4286755	0.1616459	0.3753038	0.3401733
0.70	-0.1334994	0.1503382	-0.4281569	0.1611581	0.3745440	0.3394242
0.71	-0.1334832	0.1500738	-0.4276224	0.1606561	0.3737611	0.3386420
0.72	-0.1334664	0.1498010	-0.4270709	0.1601381	0.3729517	0.3378232
0.73	-0.1334493	0.1495188	-0.4265009	0.1596022	0.3721116	0.3369642
0.74	-0.1334325	0.1492264	-0.4259109	0.1590460	0.3712357	0.3360605
0.75	-0.1334165	0.1489226	-0.4252995	0.1584665	0.3703181	0.3351071

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :-0.1336   Min.   :0.1489   Min.   :-0.4365  
 1st Qu.:0.5625   1st Qu.:-0.1336   1st Qu.:0.1507   1st Qu.:-0.4343  
 Median :0.6250   Median :-0.1336   Median :0.1521   Median :-0.4317  
 Mean   :0.6250   Mean   :-0.1335   Mean   :0.1520   Mean   :-0.4314  
 3rd Qu.:0.6875   3rd Qu.:-0.1335   3rd Qu.:0.1534   3rd Qu.:-0.4288  
 Max.   :0.7500   Max.   :-0.1334   Max.   :0.1546   Max.   :-0.4253  
     ci.ub            pvalue           sigma2      
 Min.   :0.1585   Min.   :0.3703   Min.   :0.3351  
 1st Qu.:0.1618   1st Qu.:0.3755   1st Qu.:0.3404  
 Median :0.1646   Median :0.3798   Median :0.3444  
 Mean   :0.1644   Mean   :0.3796   Mean   :0.3439  
 3rd Qu.:0.1671   3rd Qu.:0.3838   3rd Qu.:0.3478  
 Max.   :0.1694   Max.   :0.3876   Max.   :0.3506  

Prior knowledge

Code

## PRIOR KNOWLEDGE ############################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$WQP_pre <- as.factor(dataN14_control$WQP_FB_pre)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = WQP_pre)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.1465664	0.0307505	-0.2068362	-0.0862965	1.9e-06	0.1548649
0.51	-0.1470022	0.0306907	-0.2071549	-0.0868495	1.7e-06	0.1571516
0.52	-0.1474590	0.0306260	-0.2074848	-0.0874331	1.5e-06	0.1594858
0.53	-0.1479374	0.0305562	-0.2078264	-0.0880483	1.3e-06	0.1618698
0.54	-0.1484381	0.0304810	-0.2081798	-0.0886964	1.1e-06	0.1643061
0.55	-0.1489619	0.0304002	-0.2085453	-0.0893785	1.0e-06	0.1667969
0.56	-0.1495095	0.0303136	-0.2089231	-0.0900960	8.0e-07	0.1693451
0.57	-0.1500817	0.0302208	-0.2093133	-0.0908500	7.0e-07	0.1719534
0.58	-0.1506792	0.0301215	-0.2097163	-0.0916421	6.0e-07	0.1746249
0.59	-0.1513029	0.0300155	-0.2101321	-0.0924737	5.0e-07	0.1773628
0.60	-0.1519537	0.0299023	-0.2105610	-0.0933463	4.0e-07	0.1801704
0.61	-0.1526323	0.0297815	-0.2110031	-0.0942616	3.0e-07	0.1830512
0.62	-0.1533398	0.0296529	-0.2114584	-0.0952213	2.0e-07	0.1860091
0.63	-0.1540771	0.0295158	-0.2119270	-0.0962271	2.0e-07	0.1890478
0.64	-0.1548451	0.0293699	-0.2124091	-0.0972811	1.0e-07	0.1921720
0.65	-0.1556449	0.0292147	-0.2129046	-0.0983851	1.0e-07	0.1953861
0.66	-0.1564774	0.0290495	-0.2134134	-0.0995414	1.0e-07	0.1986945
0.67	-0.1573439	0.0288738	-0.2139356	-0.1007522	1.0e-07	0.2021030
0.68	-0.1582453	0.0286870	-0.2144708	-0.1020198	0.0e+00	0.2056167
0.69	-0.1591829	0.0284883	-0.2150189	-0.1033469	0.0e+00	0.2092416
0.70	-0.1601579	0.0282769	-0.2155797	-0.1047361	0.0e+00	0.2129840
0.71	-0.1611715	0.0280521	-0.2161526	-0.1061903	0.0e+00	0.2168505
0.72	-0.1622250	0.0278129	-0.2167373	-0.1077126	0.0e+00	0.2208484
0.73	-0.1633198	0.0275583	-0.2173331	-0.1093064	0.0e+00	0.2249858
0.74	-0.1644573	0.0272873	-0.2179393	-0.1109752	0.0e+00	0.2292706
0.75	-0.1656389	0.0269985	-0.2185549	-0.1127229	0.0e+00	0.2337124

Code

summary(sensitivity_moderator)

      ri_t             beta               se              ci.lb        
 Min.   :0.5000   Min.   :-0.1656   Min.   :0.02700   Min.   :-0.2186  
 1st Qu.:0.5625   1st Qu.:-0.1589   1st Qu.:0.02854   1st Qu.:-0.2149  
 Median :0.6250   Median :-0.1537   Median :0.02958   Median :-0.2117  
 Mean   :0.6250   Mean   :-0.1546   Mean   :0.02933   Mean   :-0.2121  
 3rd Qu.:0.6875   3rd Qu.:-0.1497   3rd Qu.:0.03029   3rd Qu.:-0.2090  
 Max.   :0.7500   Max.   :-0.1466   Max.   :0.03075   Max.   :-0.2068  
     ci.ub              pvalue              sigma2      
 Min.   :-0.11272   Min.   :8.509e-10   Min.   :0.1549  
 1st Qu.:-0.10302   1st Qu.:2.592e-08   1st Qu.:0.1700  
 Median :-0.09572   Median :2.057e-07   Median :0.1875  
 Mean   :-0.09707   Mean   :4.785e-07   Mean   :0.1899  
 3rd Qu.:-0.09028   3rd Qu.:7.809e-07   3rd Qu.:0.2083  
 Max.   :-0.08630   Max.   :1.876e-06   Max.   :0.2337  

Setting

Code

## SETTING ######################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$setting <- as.factor(dataN14_control$setting)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = setting)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.4974945	0.4469210	-0.3784546	1.373444	0.2656398	0.3205693
0.51	0.4965195	0.4465553	-0.3787129	1.371752	0.2661863	0.3207612
0.52	0.4955467	0.4461830	-0.3789558	1.370049	0.2667252	0.3209444
0.53	0.4945763	0.4458036	-0.3791828	1.368335	0.2672560	0.3211184
0.54	0.4936083	0.4454170	-0.3793929	1.366610	0.2677782	0.3212827
0.55	0.4926427	0.4450225	-0.3795854	1.364871	0.2682912	0.3214366
0.56	0.4916797	0.4446200	-0.3797594	1.363119	0.2687947	0.3215796
0.57	0.4907193	0.4442088	-0.3799140	1.361352	0.2692878	0.3217108
0.58	0.4897615	0.4437885	-0.3800480	1.359571	0.2697701	0.3218296
0.59	0.4888065	0.4433586	-0.3801604	1.357773	0.2702408	0.3219351
0.60	0.4878542	0.4429184	-0.3802500	1.355958	0.2706992	0.3220264
0.61	0.4869047	0.4424674	-0.3803155	1.354125	0.2711445	0.3221026
0.62	0.4859580	0.4420048	-0.3803556	1.352272	0.2715760	0.3221626
0.63	0.4850142	0.4415300	-0.3803686	1.350397	0.2719926	0.3222053
0.64	0.4840732	0.4410419	-0.3803530	1.348499	0.2723934	0.3222292
0.65	0.4831351	0.4405398	-0.3803070	1.346577	0.2727774	0.3222332
0.66	0.4821998	0.4400225	-0.3802286	1.344628	0.2731434	0.3222155
0.67	0.4812672	0.4394891	-0.3801156	1.342650	0.2734901	0.3221745
0.68	0.4803373	0.4389382	-0.3799657	1.340640	0.2738162	0.3221082
0.69	0.4794099	0.4383684	-0.3797763	1.338596	0.2741202	0.3220147
0.70	0.4784849	0.4377782	-0.3795446	1.336514	0.2744005	0.3218915
0.71	0.4775621	0.4371659	-0.3792672	1.334391	0.2746552	0.3217360
0.72	0.4766412	0.4365294	-0.3789407	1.332223	0.2748824	0.3215452
0.73	0.4757219	0.4358667	-0.3785611	1.330005	0.2750797	0.3213158
0.74	0.4748039	0.4351752	-0.3781237	1.327732	0.2752448	0.3210441
0.75	0.4738866	0.4344520	-0.3776236	1.325397	0.2753746	0.3207257

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.4739   Min.   :0.4345   Min.   :-0.3804  
 1st Qu.:0.5625   1st Qu.:0.4796   1st Qu.:0.4385   1st Qu.:-0.3802  
 Median :0.6250   Median :0.4855   Median :0.4418   Median :-0.3798  
 Mean   :0.6250   Mean   :0.4856   Mean   :0.4414   Mean   :-0.3795  
 3rd Qu.:0.6875   3rd Qu.:0.4914   3rd Qu.:0.4445   3rd Qu.:-0.3790  
 Max.   :0.7500   Max.   :0.4975   Max.   :0.4469   Max.   :-0.3776  
     ci.ub           pvalue           sigma2      
 Min.   :1.325   Min.   :0.2656   Min.   :0.3206  
 1st Qu.:1.339   1st Qu.:0.2689   1st Qu.:0.3213  
 Median :1.351   Median :0.2718   Median :0.3218  
 Mean   :1.351   Mean   :0.2713   Mean   :0.3216  
 3rd Qu.:1.363   3rd Qu.:0.2740   3rd Qu.:0.3221  
 Max.   :1.373   Max.   :0.2754   Max.   :0.3222  

Educational level

Code

## EDUCATIONAL LEVEL dummy-coded #####################
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$edu <- as.factor(dataN14_control$education_dummy)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = edu)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.4974945	0.4469210	-1.373444	0.3784546	0.2656398	0.3205693
0.51	-0.4965195	0.4465553	-1.371752	0.3787129	0.2661863	0.3207612
0.52	-0.4955467	0.4461830	-1.370049	0.3789558	0.2667252	0.3209444
0.53	-0.4945763	0.4458036	-1.368335	0.3791828	0.2672560	0.3211184
0.54	-0.4936083	0.4454170	-1.366610	0.3793929	0.2677782	0.3212827
0.55	-0.4926427	0.4450225	-1.364871	0.3795854	0.2682912	0.3214366
0.56	-0.4916797	0.4446200	-1.363119	0.3797594	0.2687947	0.3215796
0.57	-0.4907193	0.4442088	-1.361352	0.3799139	0.2692878	0.3217108
0.58	-0.4897615	0.4437885	-1.359571	0.3800480	0.2697701	0.3218296
0.59	-0.4888065	0.4433586	-1.357773	0.3801604	0.2702408	0.3219351
0.60	-0.4878542	0.4429184	-1.355958	0.3802500	0.2706992	0.3220264
0.61	-0.4869047	0.4424674	-1.354125	0.3803155	0.2711445	0.3221026
0.62	-0.4859580	0.4420048	-1.352272	0.3803556	0.2715760	0.3221626
0.63	-0.4850142	0.4415300	-1.350397	0.3803686	0.2719926	0.3222053
0.64	-0.4840732	0.4410419	-1.348499	0.3803530	0.2723934	0.3222292
0.65	-0.4831351	0.4405398	-1.346577	0.3803070	0.2727774	0.3222332
0.66	-0.4821998	0.4400225	-1.344628	0.3802286	0.2731434	0.3222155
0.67	-0.4812672	0.4394891	-1.342650	0.3801156	0.2734901	0.3221745
0.68	-0.4803373	0.4389382	-1.340640	0.3799657	0.2738162	0.3221082
0.69	-0.4794099	0.4383684	-1.338596	0.3797763	0.2741202	0.3220147
0.70	-0.4784849	0.4377782	-1.336514	0.3795446	0.2744005	0.3218915
0.71	-0.4775621	0.4371659	-1.334391	0.3792672	0.2746552	0.3217360
0.72	-0.4766412	0.4365294	-1.332223	0.3789407	0.2748824	0.3215452
0.73	-0.4757219	0.4358667	-1.330005	0.3785611	0.2750797	0.3213158
0.74	-0.4748039	0.4351752	-1.327732	0.3781237	0.2752448	0.3210441
0.75	-0.4738866	0.4344520	-1.325397	0.3776236	0.2753746	0.3207257

Code

summary(sensitivity_moderator) 

      ri_t             beta               se             ci.lb       
 Min.   :0.5000   Min.   :-0.4975   Min.   :0.4345   Min.   :-1.373  
 1st Qu.:0.5625   1st Qu.:-0.4914   1st Qu.:0.4385   1st Qu.:-1.363  
 Median :0.6250   Median :-0.4855   Median :0.4418   Median :-1.351  
 Mean   :0.6250   Mean   :-0.4856   Mean   :0.4414   Mean   :-1.351  
 3rd Qu.:0.6875   3rd Qu.:-0.4796   3rd Qu.:0.4445   3rd Qu.:-1.339  
 Max.   :0.7500   Max.   :-0.4739   Max.   :0.4469   Max.   :-1.325  
     ci.ub            pvalue           sigma2      
 Min.   :0.3776   Min.   :0.2656   Min.   :0.3206  
 1st Qu.:0.3790   1st Qu.:0.2689   1st Qu.:0.3213  
 Median :0.3798   Median :0.2718   Median :0.3218  
 Mean   :0.3795   Mean   :0.2713   Mean   :0.3216  
 3rd Qu.:0.3802   3rd Qu.:0.2740   3rd Qu.:0.3221  
 Max.   :0.3804   Max.   :0.2754   Max.   :0.3222  

Post measure

Code

##POST MEASURE##############
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$post_measure <- as.factor(dataN14_control$post_measure)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = post_measure)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.4085439	0.4784760	-1.346340	0.5292518	0.3931914	0.3661479
0.51	-0.4081858	0.4778078	-1.344672	0.5283002	0.3929450	0.3658311
0.52	-0.4078150	0.4771346	-1.342982	0.5273516	0.3927079	0.3655085
0.53	-0.4074310	0.4764562	-1.341268	0.5264060	0.3924803	0.3651795
0.54	-0.4070331	0.4757723	-1.339530	0.5254635	0.3922630	0.3648438
0.55	-0.4066206	0.4750827	-1.337765	0.5245243	0.3920562	0.3645009
0.56	-0.4061929	0.4743870	-1.335974	0.5235885	0.3918606	0.3641505
0.57	-0.4057493	0.4736849	-1.334155	0.5226560	0.3916767	0.3637919
0.58	-0.4052889	0.4729760	-1.332305	0.5217270	0.3915051	0.3634247
0.59	-0.4048109	0.4722600	-1.330423	0.5208016	0.3913464	0.3630481
0.60	-0.4043145	0.4715363	-1.328509	0.5198797	0.3912013	0.3626616
0.61	-0.4037985	0.4708045	-1.326558	0.5189614	0.3910705	0.3622644
0.62	-0.4032621	0.4700641	-1.324571	0.5180467	0.3909546	0.3618556
0.63	-0.4027040	0.4693145	-1.322544	0.5171355	0.3908546	0.3614344
0.64	-0.4021231	0.4685550	-1.320474	0.5162278	0.3907711	0.3609998
0.65	-0.4015181	0.4677849	-1.318360	0.5153234	0.3907052	0.3605506
0.66	-0.4008876	0.4670033	-1.316197	0.5144221	0.3906575	0.3600856
0.67	-0.4002300	0.4662095	-1.313984	0.5135239	0.3906293	0.3596034
0.68	-0.3995437	0.4654024	-1.311716	0.5126282	0.3906215	0.3591024
0.69	-0.3988270	0.4645811	-1.309389	0.5117352	0.3906353	0.3585815
0.70	-0.3980777	0.4637438	-1.306999	0.5108434	0.3906715	0.3580377
0.71	-0.3972939	0.4628893	-1.304540	0.5099525	0.3907316	0.3574691
0.72	-0.3964731	0.4620161	-1.302008	0.5090618	0.3908168	0.3568734
0.73	-0.3956128	0.4611223	-1.299396	0.5081703	0.3909285	0.3562477
0.74	-0.3947102	0.4602059	-1.296697	0.5072767	0.3910682	0.3555888
0.75	-0.3937619	0.4592643	-1.293903	0.5063795	0.3912374	0.3548930

Code

summary(sensitivity_moderator) 

      ri_t             beta               se             ci.lb       
 Min.   :0.5000   Min.   :-0.4085   Min.   :0.4593   Min.   :-1.346  
 1st Qu.:0.5625   1st Qu.:-0.4061   1st Qu.:0.4648   1st Qu.:-1.336  
 Median :0.6250   Median :-0.4030   Median :0.4697   Median :-1.324  
 Mean   :0.6250   Mean   :-0.4023   Mean   :0.4694   Mean   :-1.322  
 3rd Qu.:0.6875   3rd Qu.:-0.3990   3rd Qu.:0.4742   3rd Qu.:-1.310  
 Max.   :0.7500   Max.   :-0.3938   Max.   :0.4785   Max.   :-1.294  
     ci.ub            pvalue           sigma2      
 Min.   :0.5064   Min.   :0.3906   Min.   :0.3549  
 1st Qu.:0.5120   1st Qu.:0.3907   1st Qu.:0.3587  
 Median :0.5176   Median :0.3911   Median :0.3616  
 Mean   :0.5177   Mean   :0.3914   Mean   :0.3613  
 3rd Qu.:0.5234   3rd Qu.:0.3918   3rd Qu.:0.3641  
 Max.   :0.5293   Max.   :0.3932   Max.   :0.3661  

Amount

Code

## AMOUNT ######
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$amount <- as.factor(dataN14_control$amount)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = amount)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.1604791	0.4558302	-1.053890	0.7329316	0.7247940	0.3767783
0.51	-0.1602618	0.4552061	-1.052449	0.7319257	0.7247900	0.3764834
0.52	-0.1600369	0.4545774	-1.050992	0.7309185	0.7247959	0.3761818
0.53	-0.1598042	0.4539441	-1.049518	0.7299099	0.7248121	0.3758730
0.54	-0.1595632	0.4533058	-1.048026	0.7288999	0.7248390	0.3755566
0.55	-0.1593137	0.4526623	-1.046516	0.7278882	0.7248771	0.3752320
0.56	-0.1590553	0.4520134	-1.044985	0.7268747	0.7249269	0.3748989
0.57	-0.1587876	0.4513587	-1.043434	0.7258591	0.7249889	0.3745565
0.58	-0.1585102	0.4506979	-1.041862	0.7248414	0.7250636	0.3742044
0.59	-0.1582228	0.4500306	-1.040267	0.7238211	0.7251516	0.3738417
0.60	-0.1579248	0.4493566	-1.038647	0.7227979	0.7252534	0.3734678
0.61	-0.1576158	0.4486753	-1.037003	0.7217716	0.7253698	0.3730819
0.62	-0.1572952	0.4479862	-1.035332	0.7207417	0.7255012	0.3726830
0.63	-0.1569627	0.4472890	-1.033633	0.7197077	0.7256483	0.3722701
0.64	-0.1566175	0.4465830	-1.031904	0.7186690	0.7258119	0.3718422
0.65	-0.1562592	0.4458676	-1.030144	0.7176251	0.7259927	0.3713980
0.66	-0.1558871	0.4451420	-1.028350	0.7165752	0.7261913	0.3709363
0.67	-0.1555006	0.4444056	-1.026520	0.7155184	0.7264086	0.3704554
0.68	-0.1550989	0.4436575	-1.024652	0.7144538	0.7266453	0.3699537
0.69	-0.1546813	0.4428966	-1.022743	0.7133802	0.7269022	0.3694293
0.70	-0.1542470	0.4421220	-1.020790	0.7122962	0.7271802	0.3688802
0.71	-0.1537953	0.4413324	-1.018791	0.7112004	0.7274801	0.3683040
0.72	-0.1533252	0.4405264	-1.016741	0.7100908	0.7278027	0.3676980
0.73	-0.1528357	0.4397025	-1.014637	0.7089654	0.7281490	0.3670592
0.74	-0.1523259	0.4388589	-1.012474	0.7078218	0.7285197	0.3663841
0.75	-0.1517946	0.4379935	-1.010246	0.7066570	0.7289158	0.3656688

Code

summary(sensitivity_moderator) #mean = -.157, p = .726

      ri_t             beta               se             ci.lb       
 Min.   :0.5000   Min.   :-0.1605   Min.   :0.4380   Min.   :-1.054  
 1st Qu.:0.5625   1st Qu.:-0.1590   1st Qu.:0.4431   1st Qu.:-1.045  
 Median :0.6250   Median :-0.1571   Median :0.4476   Median :-1.034  
 Mean   :0.6250   Mean   :-0.1568   Mean   :0.4474   Mean   :-1.034  
 3rd Qu.:0.6875   3rd Qu.:-0.1548   3rd Qu.:0.4518   3rd Qu.:-1.023  
 Max.   :0.7500   Max.   :-0.1518   Max.   :0.4558   Max.   :-1.010  
     ci.ub            pvalue           sigma2      
 Min.   :0.7067   Min.   :0.7248   Min.   :0.3657  
 1st Qu.:0.7136   1st Qu.:0.7249   1st Qu.:0.3696  
 Median :0.7202   Median :0.7256   Median :0.3725  
 Mean   :0.7201   Mean   :0.7260   Mean   :0.3720  
 3rd Qu.:0.7266   3rd Qu.:0.7268   3rd Qu.:0.3748  
 Max.   :0.7329   Max.   :0.7289   Max.   :0.3768  

Coding

Code

## CODING #######
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$coding <- as.factor(dataN14_control$coding)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = coding)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.4845553	0.2919013	-1.056671	0.0875608	0.0969151	0.2628769
0.51	-0.4841750	0.2915151	-1.055534	0.0871841	0.0967352	0.2626270
0.52	-0.4837856	0.2911258	-1.054382	0.0868105	0.0965579	0.2623731
0.53	-0.4833867	0.2907333	-1.053214	0.0864401	0.0963832	0.2621148
0.54	-0.4829780	0.2903375	-1.052029	0.0860730	0.0962113	0.2618520
0.55	-0.4825588	0.2899381	-1.050827	0.0857094	0.0960423	0.2615844
0.56	-0.4821288	0.2895350	-1.049607	0.0853494	0.0958762	0.2613115
0.57	-0.4816873	0.2891279	-1.048368	0.0849930	0.0957133	0.2610330
0.58	-0.4812339	0.2887167	-1.047108	0.0846404	0.0955536	0.2607485
0.59	-0.4807678	0.2883010	-1.045827	0.0842918	0.0953972	0.2604576
0.60	-0.4802885	0.2878806	-1.044524	0.0839471	0.0952443	0.2601597
0.61	-0.4797952	0.2874551	-1.043197	0.0836065	0.0950949	0.2598543
0.62	-0.4792871	0.2870243	-1.041844	0.0832701	0.0949493	0.2595407
0.63	-0.4787635	0.2865876	-1.040465	0.0829378	0.0948074	0.2592183
0.64	-0.4782235	0.2861447	-1.039057	0.0826098	0.0946694	0.2588863
0.65	-0.4776661	0.2856951	-1.037618	0.0822860	0.0945355	0.2585437
0.66	-0.4770902	0.2852382	-1.036147	0.0819664	0.0944055	0.2581898
0.67	-0.4764947	0.2847734	-1.034640	0.0816509	0.0942798	0.2578232
0.68	-0.4758785	0.2843001	-1.033096	0.0813394	0.0941581	0.2574429
0.69	-0.4752402	0.2838173	-1.031512	0.0810316	0.0940407	0.2570473
0.70	-0.4745783	0.2833244	-1.029884	0.0807273	0.0939273	0.2566350
0.71	-0.4738912	0.2828201	-1.028208	0.0804260	0.0938181	0.2562040
0.72	-0.4731772	0.2823034	-1.026482	0.0801274	0.0937128	0.2557523
0.73	-0.4724344	0.2817730	-1.024699	0.0798306	0.0936112	0.2552776
0.74	-0.4716606	0.2812273	-1.022856	0.0795349	0.0935131	0.2547771
0.75	-0.4708534	0.2806646	-1.020946	0.0792392	0.0934180	0.2542477

Code

summary(sensitivity_moderator) 

      ri_t             beta               se             ci.lb       
 Min.   :0.5000   Min.   :-0.4846   Min.   :0.2807   Min.   :-1.057  
 1st Qu.:0.5625   1st Qu.:-0.4820   1st Qu.:0.2839   1st Qu.:-1.049  
 Median :0.6250   Median :-0.4790   Median :0.2868   Median :-1.041  
 Mean   :0.6250   Mean   :-0.4786   Mean   :0.2866   Mean   :-1.040  
 3rd Qu.:0.6875   3rd Qu.:-0.4754   3rd Qu.:0.2894   3rd Qu.:-1.032  
 Max.   :0.7500   Max.   :-0.4709   Max.   :0.2919   Max.   :-1.021  
     ci.ub             pvalue            sigma2      
 Min.   :0.07924   Min.   :0.09342   Min.   :0.2542  
 1st Qu.:0.08111   1st Qu.:0.09407   1st Qu.:0.2571  
 Median :0.08310   Median :0.09488   Median :0.2594  
 Mean   :0.08321   Mean   :0.09498   Mean   :0.2591  
 3rd Qu.:0.08526   3rd Qu.:0.09584   3rd Qu.:0.2612  
 Max.   :0.08756   Max.   :0.09692   Max.   :0.2629  

Level of outcome

Code

##LEVEL OF OUTCOME
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$order_outcome <- as.factor(dataN14_control$order_outcome)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = order_outcome)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.7342949	0.4483826	-0.1445189	1.613109	0.1014942	0.2561155
0.51	0.7331540	0.4480922	-0.1450905	1.611398	0.1018045	0.2563485
0.52	0.7320069	0.4477960	-0.1456571	1.609671	0.1021144	0.2565762
0.53	0.7308532	0.4474938	-0.1462184	1.607925	0.1024239	0.2567983
0.54	0.7296926	0.4471852	-0.1467744	1.606159	0.1027331	0.2570143
0.55	0.7285244	0.4468701	-0.1473249	1.604374	0.1030419	0.2572240
0.56	0.7273481	0.4465480	-0.1478699	1.602566	0.1033503	0.2574270
0.57	0.7261632	0.4462186	-0.1484093	1.600736	0.1036584	0.2576228
0.58	0.7249690	0.4458816	-0.1489429	1.598881	0.1039661	0.2578109
0.59	0.7237647	0.4455365	-0.1494708	1.597000	0.1042736	0.2579909
0.60	0.7225495	0.4451829	-0.1499930	1.595092	0.1045809	0.2581622
0.61	0.7213225	0.4448204	-0.1505094	1.593154	0.1048881	0.2583243
0.62	0.7200826	0.4444483	-0.1510201	1.591185	0.1051954	0.2584765
0.63	0.7188287	0.4440663	-0.1515252	1.589183	0.1055028	0.2586181
0.64	0.7175593	0.4436736	-0.1520250	1.587144	0.1058106	0.2587485
0.65	0.7162730	0.4432697	-0.1525196	1.585066	0.1061191	0.2588667
0.66	0.7149680	0.4428538	-0.1530094	1.582945	0.1064285	0.2589721
0.67	0.7136424	0.4424252	-0.1534949	1.580780	0.1067393	0.2590636
0.68	0.7122940	0.4419830	-0.1539768	1.578565	0.1070519	0.2591402
0.69	0.7109201	0.4415264	-0.1544557	1.576296	0.1073669	0.2592009
0.70	0.7095177	0.4410543	-0.1549329	1.573968	0.1076851	0.2592444
0.71	0.7080834	0.4405658	-0.1554097	1.571576	0.1080075	0.2592694
0.72	0.7066132	0.4400596	-0.1558877	1.569114	0.1083351	0.2592744
0.73	0.7051023	0.4395344	-0.1563693	1.566574	0.1086694	0.2592581
0.74	0.7035452	0.4389889	-0.1568572	1.563948	0.1090121	0.2592185
0.75	0.7019354	0.4384216	-0.1573551	1.561226	0.1093656	0.2591541

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.7019   Min.   :0.4384   Min.   :-0.1574  
 1st Qu.:0.5625   1st Qu.:0.7113   1st Qu.:0.4416   1st Qu.:-0.1543  
 Median :0.6250   Median :0.7195   Median :0.4443   Median :-0.1513  
 Mean   :0.6250   Mean   :0.7190   Mean   :0.4440   Mean   :-0.1511  
 3rd Qu.:0.6875   3rd Qu.:0.7271   3rd Qu.:0.4465   3rd Qu.:-0.1480  
 Max.   :0.7500   Max.   :0.7343   Max.   :0.4484   Max.   :-0.1445  
     ci.ub           pvalue           sigma2      
 Min.   :1.561   Min.   :0.1015   Min.   :0.2561  
 1st Qu.:1.577   1st Qu.:0.1034   1st Qu.:0.2575  
 Median :1.590   Median :0.1053   Median :0.2585  
 Mean   :1.589   Mean   :0.1054   Mean   :0.2582  
 3rd Qu.:1.602   3rd Qu.:0.1073   3rd Qu.:0.2592  
 Max.   :1.613   Max.   :0.1094   Max.   :0.2593  

Experiment

Code

## EXPERIMENT #########
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$experiment <- as.factor(dataN14_control$experiment)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = experiment)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.2439203	0.4827352	-1.190064	0.7022233	0.6133565	0.3587093
0.51	-0.2433421	0.4822426	-1.188520	0.7018360	0.6138361	0.3586034
0.52	-0.2427694	0.4817430	-1.186968	0.7014295	0.6143038	0.3584877
0.53	-0.2422023	0.4812361	-1.185408	0.7010030	0.6147589	0.3583619
0.54	-0.2416413	0.4807214	-1.183838	0.7005554	0.6152007	0.3582252
0.55	-0.2410864	0.4801987	-1.182259	0.7000858	0.6156285	0.3580771
0.56	-0.2405381	0.4796674	-1.180669	0.6995929	0.6160415	0.3579167
0.57	-0.2399965	0.4791272	-1.179069	0.6990755	0.6164390	0.3577436
0.58	-0.2394621	0.4785775	-1.177457	0.6985325	0.6168200	0.3575567
0.59	-0.2389352	0.4780177	-1.175833	0.6979622	0.6171836	0.3573553
0.60	-0.2384161	0.4774472	-1.174195	0.6973633	0.6175287	0.3571383
0.61	-0.2379052	0.4768655	-1.172544	0.6967340	0.6178543	0.3569048
0.62	-0.2374028	0.4762717	-1.170878	0.6960726	0.6181592	0.3566535
0.63	-0.2369095	0.4756651	-1.169196	0.6953770	0.6184421	0.3563834
0.64	-0.2364257	0.4750449	-1.167497	0.6946452	0.6187016	0.3560929
0.65	-0.2359519	0.4744100	-1.165778	0.6938747	0.6189363	0.3557805
0.66	-0.2354885	0.4737595	-1.164040	0.6930631	0.6191444	0.3554447
0.67	-0.2350362	0.4730921	-1.162280	0.6922073	0.6193241	0.3550835
0.68	-0.2345955	0.4724066	-1.160495	0.6913045	0.6194736	0.3546950
0.69	-0.2341671	0.4717016	-1.158685	0.6903510	0.6195906	0.3542768
0.70	-0.2337518	0.4709753	-1.156846	0.6893429	0.6196727	0.3538264
0.71	-0.2333502	0.4702261	-1.154977	0.6882761	0.6197174	0.3533409
0.72	-0.2329632	0.4694519	-1.153072	0.6871456	0.6197215	0.3528172
0.73	-0.2325919	0.4686504	-1.151130	0.6859460	0.6196817	0.3522516
0.74	-0.2322372	0.4678190	-1.149146	0.6846712	0.6195943	0.3516401
0.75	-0.2319005	0.4669547	-1.147115	0.6833140	0.6194549	0.3509779

Code

summary(sensitivity_moderator) 

      ri_t             beta               se             ci.lb       
 Min.   :0.5000   Min.   :-0.2439   Min.   :0.4670   Min.   :-1.190  
 1st Qu.:0.5625   1st Qu.:-0.2404   1st Qu.:0.4719   1st Qu.:-1.180  
 Median :0.6250   Median :-0.2372   Median :0.4760   Median :-1.170  
 Mean   :0.6250   Mean   :-0.2374   Mean   :0.4756   Mean   :-1.170  
 3rd Qu.:0.6875   3rd Qu.:-0.2343   3rd Qu.:0.4795   3rd Qu.:-1.159  
 Max.   :0.7500   Max.   :-0.2319   Max.   :0.4827   Max.   :-1.147  
     ci.ub            pvalue           sigma2      
 Min.   :0.6833   Min.   :0.6134   Min.   :0.3510  
 1st Qu.:0.6906   1st Qu.:0.6161   1st Qu.:0.3544  
 Median :0.6957   Median :0.6183   Median :0.3565  
 Mean   :0.6947   Mean   :0.6176   Mean   :0.3559  
 3rd Qu.:0.6995   3rd Qu.:0.6195   3rd Qu.:0.3579  
 Max.   :0.7022   Max.   :0.6197   Max.   :0.3587  

System

Code

## SYSTEM ############
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$system <- as.factor(dataN14_control$system_type)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = system)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.2015433	0.1904852	-0.5748875	0.1718009	0.2900316	0.2887986
0.51	-0.2011077	0.1903924	-0.5742700	0.1720546	0.2908403	0.2889952
0.52	-0.2006746	0.1902960	-0.5736480	0.1722987	0.2916361	0.2891813
0.53	-0.2002441	0.1901958	-0.5730211	0.1725329	0.2924183	0.2893561
0.54	-0.1998163	0.1900916	-0.5723891	0.1727564	0.2931861	0.2895191
0.55	-0.1993914	0.1899833	-0.5717517	0.1729689	0.2939385	0.2896695
0.56	-0.1989694	0.1898704	-0.5711087	0.1731698	0.2946745	0.2898067
0.57	-0.1985506	0.1897530	-0.5704596	0.1733584	0.2953932	0.2899297
0.58	-0.1981352	0.1896304	-0.5698041	0.1735336	0.2960927	0.2900373
0.59	-0.1977232	0.1895028	-0.5691419	0.1736956	0.2967731	0.2901296
0.60	-0.1973147	0.1893697	-0.5684726	0.1738431	0.2974325	0.2902048
0.61	-0.1969102	0.1892307	-0.5677956	0.1739752	0.2980694	0.2902621
0.62	-0.1965097	0.1890855	-0.5671105	0.1740911	0.2986822	0.2903001
0.63	-0.1961134	0.1889337	-0.5664166	0.1741897	0.2992693	0.2903175
0.64	-0.1957217	0.1887747	-0.5657134	0.1742699	0.2998289	0.2903128
0.65	-0.1953348	0.1886082	-0.5650000	0.1743305	0.3003588	0.2902844
0.66	-0.1949529	0.1884336	-0.5642758	0.1743701	0.3008570	0.2902306
0.67	-0.1945763	0.1882502	-0.5635399	0.1743873	0.3013209	0.2901493
0.68	-0.1942055	0.1880574	-0.5627912	0.1743803	0.3017478	0.2900384
0.69	-0.1938407	0.1878545	-0.5620288	0.1743473	0.3021348	0.2898954
0.70	-0.1934824	0.1876406	-0.5612512	0.1742864	0.3024786	0.2897176
0.71	-0.1931310	0.1874148	-0.5604572	0.1741953	0.3027753	0.2895019
0.72	-0.1927869	0.1871760	-0.5596451	0.1740713	0.3030209	0.2892449
0.73	-0.1924508	0.1869230	-0.5588132	0.1739115	0.3032106	0.2889428
0.74	-0.1921233	0.1866545	-0.5579594	0.1737128	0.3033391	0.2885912
0.75	-0.1918051	0.1863689	-0.5570815	0.1734712	0.3034002	0.2881849

Code

summary(sensitivity_moderator) 

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :-0.2015   Min.   :0.1864   Min.   :-0.5749  
 1st Qu.:0.5625   1st Qu.:-0.1989   1st Qu.:0.1879   1st Qu.:-0.5709  
 Median :0.6250   Median :-0.1963   Median :0.1890   Median :-0.5668  
 Mean   :0.6250   Mean   :-0.1964   Mean   :0.1888   Mean   :-0.5665  
 3rd Qu.:0.6875   3rd Qu.:-0.1939   3rd Qu.:0.1898   3rd Qu.:-0.5622  
 Max.   :0.7500   Max.   :-0.1918   Max.   :0.1905   Max.   :-0.5571  
     ci.ub            pvalue           sigma2      
 Min.   :0.1718   Min.   :0.2900   Min.   :0.2882  
 1st Qu.:0.1732   1st Qu.:0.2949   1st Qu.:0.2893  
 Median :0.1739   Median :0.2990   Median :0.2899  
 Mean   :0.1736   Mean   :0.2982   Mean   :0.2897  
 3rd Qu.:0.1743   3rd Qu.:0.3020   3rd Qu.:0.2902  
 Max.   :0.1744   Max.   :0.3034   Max.   :0.2903  

Teacher effects

Code

## TEACHER EFFECTS ########
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$teacher <- as.factor(dataN14_control$teacher_effects)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = teacher)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.3780995	0.2389033	-0.8463414	0.0901423	0.1135020	0.2239991
0.51	-0.3773235	0.2387537	-0.8452721	0.0906251	0.1140180	0.2242824
0.52	-0.3765474	0.2385991	-0.8441931	0.0910983	0.1145291	0.2245574
0.53	-0.3757712	0.2384395	-0.8431040	0.0915616	0.1150349	0.2248239
0.54	-0.3749949	0.2382745	-0.8420042	0.0920145	0.1155351	0.2250812
0.55	-0.3742184	0.2381038	-0.8408933	0.0924565	0.1160293	0.2253288
0.56	-0.3734417	0.2379273	-0.8397706	0.0928871	0.1165169	0.2255661
0.57	-0.3726649	0.2377445	-0.8386355	0.0933058	0.1169976	0.2257925
0.58	-0.3718877	0.2375552	-0.8374874	0.0937120	0.1174707	0.2260073
0.59	-0.3711102	0.2373590	-0.8363253	0.0941049	0.1179357	0.2262097
0.60	-0.3703323	0.2371555	-0.8351486	0.0944839	0.1183919	0.2263989
0.61	-0.3695540	0.2369443	-0.8339562	0.0948482	0.1188387	0.2265740
0.62	-0.3687752	0.2367248	-0.8327473	0.0951968	0.1192753	0.2267340
0.63	-0.3679958	0.2364966	-0.8315206	0.0955289	0.1197009	0.2268777
0.64	-0.3672158	0.2362590	-0.8302751	0.0958434	0.1201145	0.2270041
0.65	-0.3664351	0.2360116	-0.8290093	0.0961390	0.1205151	0.2271117
0.66	-0.3656536	0.2357534	-0.8277217	0.0964145	0.1209016	0.2271990
0.67	-0.3648712	0.2354838	-0.8264109	0.0966685	0.1212727	0.2272645
0.68	-0.3640876	0.2352020	-0.8250750	0.0968998	0.1216274	0.2273067
0.69	-0.3633032	0.2349067	-0.8237117	0.0971054	0.1219633	0.2273227
0.70	-0.3625175	0.2345969	-0.8223189	0.0972839	0.1222789	0.2273106
0.71	-0.3617305	0.2342714	-0.8208940	0.0974329	0.1225723	0.2272678
0.72	-0.3609422	0.2339288	-0.8194342	0.0975497	0.1228410	0.2271914
0.73	-0.3601525	0.2335674	-0.8179361	0.0976312	0.1230824	0.2270780
0.74	-0.3593612	0.2331853	-0.8163960	0.0976737	0.1232932	0.2269238
0.75	-0.3585684	0.2327805	-0.8148098	0.0976730	0.1234699	0.2267244

Code

summary(sensitivity_moderator) 

      ri_t             beta               se             ci.lb        
 Min.   :0.5000   Min.   :-0.3781   Min.   :0.2328   Min.   :-0.8463  
 1st Qu.:0.5625   1st Qu.:-0.3732   1st Qu.:0.2350   1st Qu.:-0.8395  
 Median :0.6250   Median :-0.3684   Median :0.2366   Median :-0.8321  
 Mean   :0.6250   Mean   :-0.3684   Mean   :0.2363   Mean   :-0.8316  
 3rd Qu.:0.6875   3rd Qu.:-0.3635   3rd Qu.:0.2379   3rd Qu.:-0.8241  
 Max.   :0.7500   Max.   :-0.3586   Max.   :0.2389   Max.   :-0.8148  
     ci.ub             pvalue           sigma2      
 Min.   :0.09014   Min.   :0.1135   Min.   :0.2240  
 1st Qu.:0.09299   1st Qu.:0.1166   1st Qu.:0.2256  
 Median :0.09536   Median :0.1195   Median :0.2267  
 Mean   :0.09486   Mean   :0.1191   Mean   :0.2263  
 3rd Qu.:0.09705   3rd Qu.:0.1219   3rd Qu.:0.2272  
 Max.   :0.09767   Max.   :0.1235   Max.   :0.2273  

Validated tool

Code

## VALIDATED TOOL ###############
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$validated_tool <- as.factor(dataN14_control$validated_tool)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = validated_tool)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	1.114876	0.4352933	0.2617172	1.968036	0.0104309	0.1519381
0.51	1.114375	0.4349179	0.2619516	1.966798	0.0103992	0.1518810
0.52	1.113866	0.4345358	0.2621918	1.965541	0.0103668	0.1518193
0.53	1.113350	0.4341465	0.2624383	1.964261	0.0103338	0.1517529
0.54	1.112825	0.4337496	0.2626917	1.962959	0.0102999	0.1516812
0.55	1.112292	0.4333444	0.2629527	1.961632	0.0102653	0.1516040
0.56	1.111750	0.4329305	0.2632221	1.960278	0.0102298	0.1515208
0.57	1.111199	0.4325070	0.2635006	1.958897	0.0101932	0.1514310
0.58	1.110638	0.4320738	0.2637886	1.957486	0.0101557	0.1513346
0.59	1.110066	0.4316291	0.2640884	1.956043	0.0101170	0.1512303
0.60	1.109483	0.4311725	0.2644007	1.954566	0.0100770	0.1511177
0.61	1.108889	0.4307031	0.2647266	1.953052	0.0100355	0.1509963
0.62	1.108283	0.4302197	0.2650677	1.951498	0.0099926	0.1508652
0.63	1.107663	0.4297211	0.2654257	1.949901	0.0099479	0.1507235
0.64	1.107031	0.4292059	0.2658023	1.948259	0.0099014	0.1505703
0.65	1.106383	0.4286727	0.2661999	1.946566	0.0098529	0.1504045
0.66	1.105720	0.4281197	0.2666208	1.944819	0.0098020	0.1502249
0.67	1.105041	0.4275451	0.2670678	1.943014	0.0097487	0.1500301
0.68	1.104344	0.4269463	0.2675449	1.941143	0.0096925	0.1498184
0.69	1.103629	0.4263216	0.2680540	1.939204	0.0096332	0.1495886
0.70	1.102894	0.4256679	0.2686002	1.937188	0.0095705	0.1493385
0.71	1.102138	0.4249821	0.2691883	1.935088	0.0095039	0.1490659
0.72	1.101359	0.4242607	0.2698239	1.932895	0.0094330	0.1487682
0.73	1.100557	0.4234994	0.2705133	1.930600	0.0093572	0.1484427
0.74	1.099729	0.4226936	0.2712643	1.928193	0.0092759	0.1480860
0.75	1.098872	0.4218378	0.2720857	1.925659	0.0091884	0.1476943

Code

summary(sensitivity_moderator) 

      ri_t             beta             se             ci.lb       
 Min.   :0.5000   Min.   :1.099   Min.   :0.4218   Min.   :0.2617  
 1st Qu.:0.5625   1st Qu.:1.104   1st Qu.:0.4265   1st Qu.:0.2633  
 Median :0.6250   Median :1.108   Median :0.4300   Median :0.2652  
 Mean   :0.6250   Mean   :1.108   Mean   :0.4295   Mean   :0.2658  
 3rd Qu.:0.6875   3rd Qu.:1.112   3rd Qu.:0.4328   3rd Qu.:0.2679  
 Max.   :0.7500   Max.   :1.115   Max.   :0.4353   Max.   :0.2721  
     ci.ub           pvalue             sigma2      
 Min.   :1.926   Min.   :0.009188   Min.   :0.1477  
 1st Qu.:1.940   1st Qu.:0.009648   1st Qu.:0.1496  
 Median :1.951   Median :0.009970   Median :0.1508  
 Mean   :1.949   Mean   :0.009916   Mean   :0.1505  
 3rd Qu.:1.960   3rd Qu.:0.010221   3rd Qu.:0.1515  
 Max.   :1.968   Max.   :0.010431   Max.   :0.1519  

Reliability of measure

Code

## RELIABILITY MEASURE ###########
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$reliability <- as.factor(dataN14_control$reliability_measurement)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = reliability)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.6580934	0.2236447	-1.096429	-0.2197579	0.0032548	0.1374712
0.51	-0.6568891	0.2241825	-1.096279	-0.2174996	0.0033879	0.1385670
0.52	-0.6556745	0.2247178	-1.096113	-0.2152356	0.0035255	0.1396651
0.53	-0.6544488	0.2252503	-1.095931	-0.2129664	0.0036675	0.1407648
0.54	-0.6532114	0.2257803	-1.095733	-0.2106901	0.0038143	0.1418669
0.55	-0.6519615	0.2263078	-1.095517	-0.2084064	0.0039659	0.1429710
0.56	-0.6506982	0.2268327	-1.095282	-0.2061142	0.0041226	0.1440771
0.57	-0.6494207	0.2273552	-1.095029	-0.2038127	0.0042845	0.1451853
0.58	-0.6481277	0.2278751	-1.094755	-0.2015008	0.0044520	0.1462955
0.59	-0.6468184	0.2283925	-1.094459	-0.1991773	0.0046251	0.1474079
0.60	-0.6454914	0.2289075	-1.094142	-0.1968410	0.0048042	0.1485223
0.61	-0.6441453	0.2294200	-1.093800	-0.1944903	0.0049895	0.1496388
0.62	-0.6427787	0.2299302	-1.093434	-0.1921237	0.0051813	0.1507574
0.63	-0.6413899	0.2304381	-1.093040	-0.1897395	0.0053801	0.1518782
0.64	-0.6399771	0.2309437	-1.092618	-0.1873357	0.0055861	0.1530013
0.65	-0.6385381	0.2314471	-1.092166	-0.1849102	0.0057997	0.1541267
0.66	-0.6370708	0.2319484	-1.091681	-0.1824603	0.0060215	0.1552546
0.67	-0.6355726	0.2324477	-1.091162	-0.1799835	0.0062520	0.1563850
0.68	-0.6340405	0.2329450	-1.090604	-0.1774766	0.0064918	0.1575181
0.69	-0.6324713	0.2334408	-1.090007	-0.1749357	0.0067417	0.1586544
0.70	-0.6308613	0.2339349	-1.089365	-0.1723573	0.0070022	0.1597937
0.71	-0.6292064	0.2344277	-1.088676	-0.1697366	0.0072745	0.1609365
0.72	-0.6275016	0.2349193	-1.087935	-0.1670682	0.0075595	0.1620833
0.73	-0.6257416	0.2354103	-1.087137	-0.1643458	0.0078585	0.1632346
0.74	-0.6239199	0.2359011	-1.086278	-0.1615622	0.0081730	0.1643911
0.75	-0.6220291	0.2363922	-1.085349	-0.1587089	0.0085048	0.1655538

Code

summary(sensitivity_moderator) 

      ri_t             beta               se             ci.lb       
 Min.   :0.5000   Min.   :-0.6581   Min.   :0.2236   Min.   :-1.096  
 1st Qu.:0.5625   1st Qu.:-0.6504   1st Qu.:0.2270   1st Qu.:-1.095  
 Median :0.6250   Median :-0.6421   Median :0.2302   Median :-1.093  
 Mean   :0.6250   Mean   :-0.6414   Mean   :0.2301   Mean   :-1.092  
 3rd Qu.:0.6875   3rd Qu.:-0.6329   3rd Qu.:0.2333   3rd Qu.:-1.090  
 Max.   :0.7500   Max.   :-0.6220   Max.   :0.2364   Max.   :-1.085  
     ci.ub             pvalue             sigma2      
 Min.   :-0.2198   Min.   :0.003255   Min.   :0.1375  
 1st Qu.:-0.2055   1st Qu.:0.004163   1st Qu.:0.1444  
 Median :-0.1909   Median :0.005281   Median :0.1513  
 Mean   :-0.1904   Mean   :0.005489   Mean   :0.1514  
 3rd Qu.:-0.1756   3rd Qu.:0.006679   3rd Qu.:0.1584  
 Max.   :-0.1587   Max.   :0.008505   Max.   :0.1656  

Code

#post-hoc:
#low
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$reliability_low <- ifelse(dataN14_control$reliability_measurement == 0, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = reliability_low)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.8935590	0.5604653	-0.2049329	1.992051	0.1108652	0.2626922
0.51	0.8926921	0.5604341	-0.2057386	1.991123	0.1111920	0.2628427
0.52	0.8918096	0.5603991	-0.2065525	1.990172	0.1115234	0.2629897
0.53	0.8909106	0.5603602	-0.2073752	1.989196	0.1118598	0.2631331
0.54	0.8899938	0.5603170	-0.2082074	1.988195	0.1122015	0.2632726
0.55	0.8890581	0.5602695	-0.2090500	1.987166	0.1125489	0.2634081
0.56	0.8881021	0.5602174	-0.2099039	1.986108	0.1129025	0.2635394
0.57	0.8871243	0.5601604	-0.2107699	1.985019	0.1132627	0.2636661
0.58	0.8861232	0.5600983	-0.2116493	1.983896	0.1136301	0.2637881
0.59	0.8850969	0.5600307	-0.2125431	1.982737	0.1140053	0.2639051
0.60	0.8840434	0.5599574	-0.2134530	1.981540	0.1143891	0.2640168
0.61	0.8829607	0.5598781	-0.2143803	1.980302	0.1147820	0.2641228
0.62	0.8818461	0.5597924	-0.2153269	1.979019	0.1151851	0.2642230
0.63	0.8806971	0.5597000	-0.2162947	1.977689	0.1155993	0.2643168
0.64	0.8795105	0.5596004	-0.2172861	1.976307	0.1160257	0.2644040
0.65	0.8782831	0.5594932	-0.2183035	1.974870	0.1164656	0.2644841
0.66	0.8770108	0.5593780	-0.2193500	1.973372	0.1169203	0.2645568
0.67	0.8756895	0.5592543	-0.2204288	1.971808	0.1173916	0.2646215
0.68	0.8743143	0.5591217	-0.2215440	1.970173	0.1178814	0.2646779
0.69	0.8728796	0.5589794	-0.2227000	1.968459	0.1183918	0.2647253
0.70	0.8713791	0.5588271	-0.2239019	1.966660	0.1189254	0.2647633
0.71	0.8698055	0.5586640	-0.2251559	1.964767	0.1194851	0.2647912
0.72	0.8681505	0.5584896	-0.2264691	1.962770	0.1200744	0.2648086
0.73	0.8664044	0.5583027	-0.2278488	1.960658	0.1206971	0.2648143
0.74	0.8645561	0.5581036	-0.2293069	1.958419	0.1213586	0.2648087
0.75	0.8625924	0.5578911	-0.2308541	1.956039	0.1220644	0.2647906

Code

summary(sensitivity_moderator) #mean = .88, p = .116

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.8626   Min.   :0.5579   Min.   :-0.2309  
 1st Qu.:0.5625   1st Qu.:0.8732   1st Qu.:0.5590   1st Qu.:-0.2224  
 Median :0.6250   Median :0.8813   Median :0.5597   Median :-0.2158  
 Mean   :0.6250   Mean   :0.8802   Mean   :0.5595   Mean   :-0.2165  
 3rd Qu.:0.6875   3rd Qu.:0.8879   3rd Qu.:0.5602   3rd Qu.:-0.2101  
 Max.   :0.7500   Max.   :0.8936   Max.   :0.5605   Max.   :-0.2049  
     ci.ub           pvalue           sigma2      
 Min.   :1.956   Min.   :0.1109   Min.   :0.2627  
 1st Qu.:1.969   1st Qu.:0.1130   1st Qu.:0.2636  
 Median :1.978   Median :0.1154   Median :0.2643  
 Mean   :1.977   Mean   :0.1158   Mean   :0.2641  
 3rd Qu.:1.986   3rd Qu.:0.1183   3rd Qu.:0.2647  
 Max.   :1.992   Max.   :0.1221   Max.   :0.2648  

Code

#high
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$reliability_high <- ifelse(dataN14_control$reliability_measurement == 1, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = reliability_high)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.3136996	0.4742084	-0.6157317	1.243131	0.5082771	0.3845022
0.51	0.3133977	0.4736375	-0.6149147	1.241710	0.5081747	0.3842278
0.52	0.3130869	0.4730617	-0.6140969	1.240271	0.5080795	0.3839466
0.53	0.3127670	0.4724807	-0.6132781	1.238812	0.5079918	0.3836582
0.54	0.3124376	0.4718942	-0.6124581	1.237333	0.5079120	0.3833621
0.55	0.3120982	0.4713020	-0.6116366	1.235833	0.5078402	0.3830579
0.56	0.3117485	0.4707036	-0.6108136	1.234311	0.5077769	0.3827449
0.57	0.3113880	0.4700988	-0.6099887	1.232765	0.5077223	0.3824226
0.58	0.3110163	0.4694871	-0.6091616	1.231194	0.5076767	0.3820903
0.59	0.3106327	0.4688681	-0.6083319	1.229597	0.5076405	0.3817472
0.60	0.3102368	0.4682414	-0.6074994	1.227973	0.5076141	0.3813927
0.61	0.3098279	0.4676063	-0.6066636	1.226319	0.5075978	0.3810258
0.62	0.3094055	0.4669624	-0.6058239	1.224635	0.5075919	0.3806456
0.63	0.3089689	0.4663090	-0.6049799	1.222918	0.5075970	0.3802510
0.64	0.3085174	0.4656454	-0.6041308	1.221166	0.5076133	0.3798408
0.65	0.3080501	0.4649709	-0.6032760	1.219376	0.5076413	0.3794139
0.66	0.3075664	0.4642845	-0.6024145	1.217547	0.5076813	0.3789687
0.67	0.3070652	0.4635854	-0.6015455	1.215676	0.5077339	0.3785037
0.68	0.3065457	0.4628725	-0.6006678	1.213759	0.5077994	0.3780171
0.69	0.3060067	0.4621446	-0.5997801	1.211794	0.5078783	0.3775070
0.70	0.3054473	0.4614004	-0.5988810	1.209775	0.5079710	0.3769711
0.71	0.3048660	0.4606385	-0.5979688	1.207701	0.5080779	0.3764070
0.72	0.3042617	0.4598573	-0.5970420	1.205566	0.5081995	0.3758125
0.73	0.3036328	0.4590545	-0.5960974	1.203363	0.5083361	0.3751831
0.74	0.3029776	0.4582281	-0.5951330	1.201088	0.5084880	0.3745160
0.75	0.3022943	0.4573757	-0.5941456	1.198734	0.5086559	0.3738070

Code

summary(sensitivity_moderator) #mean = .31, p = .508

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.3023   Min.   :0.4574   Min.   :-0.6157  
 1st Qu.:0.5625   1st Qu.:0.3061   1st Qu.:0.4623   1st Qu.:-0.6106  
 Median :0.6250   Median :0.3092   Median :0.4666   Median :-0.6054  
 Mean   :0.6250   Mean   :0.3088   Mean   :0.4663   Mean   :-0.6052  
 3rd Qu.:0.6875   3rd Qu.:0.3117   3rd Qu.:0.4706   3rd Qu.:-0.6000  
 Max.   :0.7500   Max.   :0.3137   Max.   :0.4742   Max.   :-0.5941  
     ci.ub           pvalue           sigma2      
 Min.   :1.199   Min.   :0.5076   Min.   :0.3738  
 1st Qu.:1.212   1st Qu.:0.5077   1st Qu.:0.3776  
 Median :1.224   Median :0.5078   Median :0.3804  
 Mean   :1.223   Mean   :0.5079   Mean   :0.3800  
 3rd Qu.:1.234   3rd Qu.:0.5081   3rd Qu.:0.3827  
 Max.   :1.243   Max.   :0.5087   Max.   :0.3845  

Code

#not specified
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$reliability_not <- ifelse(dataN14_control$reliability_measurement == 2, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = reliability_not)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.8332949	0.3858626	-1.589572	-0.0770181	0.0308064	0.2086035
0.51	-0.8313473	0.3858681	-1.587635	-0.0750597	0.0312021	0.2093181
0.52	-0.8293853	0.3858724	-1.585681	-0.0730892	0.0316045	0.2100339
0.53	-0.8274080	0.3858752	-1.583710	-0.0711065	0.0320139	0.2107506
0.54	-0.8254147	0.3858763	-1.581718	-0.0691111	0.0324304	0.2114680
0.55	-0.8234043	0.3858755	-1.579706	-0.0671022	0.0328542	0.2121859
0.56	-0.8213760	0.3858729	-1.577673	-0.0650791	0.0332859	0.2129042
0.57	-0.8193285	0.3858681	-1.575616	-0.0630409	0.0337255	0.2136226
0.58	-0.8172607	0.3858611	-1.573535	-0.0609868	0.0341736	0.2143409
0.59	-0.8151712	0.3858517	-1.571427	-0.0589158	0.0346304	0.2150590
0.60	-0.8130586	0.3858396	-1.569290	-0.0568268	0.0350964	0.2157766
0.61	-0.8109214	0.3858248	-1.567124	-0.0547187	0.0355720	0.2164933
0.62	-0.8087577	0.3858069	-1.564925	-0.0525901	0.0360577	0.2172090
0.63	-0.8065657	0.3857857	-1.562692	-0.0504396	0.0365541	0.2179233
0.64	-0.8043432	0.3857609	-1.560421	-0.0482656	0.0370619	0.2186357
0.65	-0.8020879	0.3857323	-1.558109	-0.0460664	0.0375815	0.2193461
0.66	-0.7997972	0.3856995	-1.555754	-0.0438400	0.0381139	0.2200538
0.67	-0.7974681	0.3856622	-1.553352	-0.0415842	0.0386598	0.2207585
0.68	-0.7950975	0.3856199	-1.550898	-0.0392964	0.0392203	0.2214596
0.69	-0.7926815	0.3855722	-1.548389	-0.0369739	0.0397963	0.2221565
0.70	-0.7902162	0.3855187	-1.545819	-0.0346135	0.0403890	0.2228488
0.71	-0.7876969	0.3854588	-1.543182	-0.0322114	0.0410000	0.2235355
0.72	-0.7851181	0.3853920	-1.540473	-0.0297636	0.0416307	0.2242162
0.73	-0.7824739	0.3853177	-1.537683	-0.0272651	0.0422830	0.2248898
0.74	-0.7797571	0.3852351	-1.534804	-0.0247102	0.0429591	0.2255556
0.75	-0.7769598	0.3851436	-1.531827	-0.0220921	0.0436616	0.2262127

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb       
 Min.   :0.5000   Min.   :-0.8333   Min.   :0.3851   Min.   :-1.590  
 1st Qu.:0.5625   1st Qu.:-0.8209   1st Qu.:0.3856   1st Qu.:-1.577  
 Median :0.6250   Median :-0.8077   Median :0.3858   Median :-1.564  
 Mean   :0.6250   Mean   :-0.8068   Mean   :0.3857   Mean   :-1.563  
 3rd Qu.:0.6875   3rd Qu.:-0.7933   3rd Qu.:0.3859   3rd Qu.:-1.549  
 Max.   :0.7500   Max.   :-0.7770   Max.   :0.3859   Max.   :-1.532  
     ci.ub              pvalue            sigma2      
 Min.   :-0.07702   Min.   :0.03081   Min.   :0.2086  
 1st Qu.:-0.06457   1st Qu.:0.03340   1st Qu.:0.2131  
 Median :-0.05151   Median :0.03631   Median :0.2176  
 Mean   :-0.05084   Mean   :0.03663   Mean   :0.2175  
 3rd Qu.:-0.03755   3rd Qu.:0.03965   3rd Qu.:0.2220  
 Max.   :-0.02209   Max.   :0.04366   Max.   :0.2262  

Treatment fidelity

Code

## TREATMENT FIDELITY ###########
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$fidelity <- as.factor(dataN14_control$treatment_fidelity)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = fidelity)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.4845101	0.1776556	0.1363115	0.8327087	0.0063866	0.1589562
0.51	0.4839456	0.1775103	0.1360319	0.8318593	0.0064049	0.1591700
0.52	0.4833787	0.1773619	0.1357558	0.8310016	0.0064227	0.1593789
0.53	0.4828092	0.1772102	0.1354835	0.8301349	0.0064399	0.1595825
0.54	0.4822370	0.1770551	0.1352154	0.8292586	0.0064564	0.1597804
0.55	0.4816618	0.1768965	0.1349510	0.8283726	0.0064722	0.1599726
0.56	0.4810834	0.1767341	0.1346909	0.8274760	0.0064874	0.1601585
0.57	0.4805016	0.1765678	0.1344350	0.8265682	0.0065017	0.1603378
0.58	0.4799161	0.1763974	0.1341836	0.8256486	0.0065153	0.1605099
0.59	0.4793266	0.1762225	0.1339368	0.8247164	0.0065280	0.1606744
0.60	0.4787327	0.1760430	0.1336947	0.8237707	0.0065399	0.1608308
0.61	0.4781341	0.1758586	0.1334575	0.8228106	0.0065508	0.1609785
0.62	0.4775302	0.1756690	0.1332254	0.8218351	0.0065608	0.1611169
0.63	0.4769207	0.1754738	0.1329984	0.8208431	0.0065697	0.1612455
0.64	0.4763050	0.1752728	0.1327767	0.8198334	0.0065776	0.1613634
0.65	0.4756825	0.1750655	0.1325605	0.8188045	0.0065843	0.1614699
0.66	0.4750525	0.1748515	0.1323499	0.8177551	0.0065898	0.1615642
0.67	0.4744142	0.1746304	0.1321449	0.8166835	0.0065941	0.1616453
0.68	0.4737667	0.1744017	0.1319457	0.8155877	0.0065971	0.1617122
0.69	0.4731089	0.1741648	0.1317523	0.8144656	0.0065987	0.1617640
0.70	0.4724399	0.1739191	0.1315646	0.8133151	0.0065989	0.1617992
0.71	0.4717580	0.1736641	0.1313827	0.8121334	0.0065977	0.1618167
0.72	0.4710620	0.1733989	0.1312064	0.8109175	0.0065949	0.1618150
0.73	0.4703499	0.1731227	0.1310356	0.8096642	0.0065905	0.1617924
0.74	0.4696196	0.1728347	0.1308698	0.8083695	0.0065845	0.1617473
0.75	0.4688689	0.1725340	0.1307085	0.8070292	0.0065768	0.1616776

Code

summary(sensitivity_moderator) 

      ri_t             beta              se             ci.lb       
 Min.   :0.5000   Min.   :0.4689   Min.   :0.1725   Min.   :0.1307  
 1st Qu.:0.5625   1st Qu.:0.4733   1st Qu.:0.1742   1st Qu.:0.1318  
 Median :0.6250   Median :0.4772   Median :0.1756   Median :0.1331  
 Mean   :0.6250   Mean   :0.4770   Mean   :0.1754   Mean   :0.1333  
 3rd Qu.:0.6875   3rd Qu.:0.4809   3rd Qu.:0.1767   3rd Qu.:0.1346  
 Max.   :0.7500   Max.   :0.4845   Max.   :0.1777   Max.   :0.1363  
     ci.ub            pvalue             sigma2      
 Min.   :0.8070   Min.   :0.006387   Min.   :0.1590  
 1st Qu.:0.8147   1st Qu.:0.006491   1st Qu.:0.1602  
 Median :0.8213   Median :0.006565   Median :0.1612  
 Mean   :0.8208   Mean   :0.006535   Mean   :0.1609  
 3rd Qu.:0.8272   3rd Qu.:0.006590   3rd Qu.:0.1617  
 Max.   :0.8327   Max.   :0.006599   Max.   :0.1618  

Code

#post-hoc:
#not specified
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$fidelity_not <- ifelse(dataN14_control$treatment_fidelity == 0, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = fidelity_not)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	-0.9490237	0.3387597	-1.612980	-0.2850669	0.0050871	0.1550865
0.51	-0.9481592	0.3383239	-1.611262	-0.2850564	0.0050705	0.1551256
0.52	-0.9472939	0.3378794	-1.609525	-0.2850626	0.0050528	0.1551565
0.53	-0.9464279	0.3374255	-1.607770	-0.2850861	0.0050340	0.1551787
0.54	-0.9455609	0.3369619	-1.605994	-0.2851278	0.0050140	0.1551916
0.55	-0.9446929	0.3364881	-1.604197	-0.2851884	0.0049926	0.1551949
0.56	-0.9438236	0.3360034	-1.602378	-0.2852690	0.0049700	0.1551878
0.57	-0.9429529	0.3355074	-1.600535	-0.2853704	0.0049460	0.1551699
0.58	-0.9420805	0.3349994	-1.598667	-0.2854937	0.0049206	0.1551404
0.59	-0.9412062	0.3344785	-1.596772	-0.2856404	0.0048937	0.1550985
0.60	-0.9403299	0.3339444	-1.594849	-0.2858108	0.0048652	0.1550437
0.61	-0.9394510	0.3333960	-1.592895	-0.2860068	0.0048350	0.1549751
0.62	-0.9385694	0.3328325	-1.590909	-0.2862296	0.0048032	0.1548917
0.63	-0.9376845	0.3322529	-1.588888	-0.2864808	0.0047695	0.1547925
0.64	-0.9367961	0.3316561	-1.586830	-0.2867620	0.0047340	0.1546765
0.65	-0.9359034	0.3310410	-1.584732	-0.2870749	0.0046964	0.1545425
0.66	-0.9350061	0.3304063	-1.582591	-0.2874216	0.0046567	0.1543890
0.67	-0.9341033	0.3297506	-1.580403	-0.2878040	0.0046148	0.1542148
0.68	-0.9331944	0.3290723	-1.578164	-0.2882246	0.0045706	0.1540182
0.69	-0.9322785	0.3283696	-1.575871	-0.2886858	0.0045239	0.1537974
0.70	-0.9313545	0.3276407	-1.573519	-0.2891905	0.0044746	0.1535505
0.71	-0.9304214	0.3268833	-1.571101	-0.2897418	0.0044225	0.1532753
0.72	-0.9294777	0.3260950	-1.568612	-0.2903432	0.0043674	0.1529693
0.73	-0.9285221	0.3252731	-1.566046	-0.2909986	0.0043092	0.1526298
0.74	-0.9275528	0.3244144	-1.563393	-0.2917123	0.0042476	0.1522536
0.75	-0.9265677	0.3235154	-1.560646	-0.2924893	0.0041825	0.1518372

Code

summary(sensitivity_moderator) #mean = -.94, p = .005

      ri_t             beta               se             ci.lb       
 Min.   :0.5000   Min.   :-0.9490   Min.   :0.3235   Min.   :-1.613  
 1st Qu.:0.5625   1st Qu.:-0.9436   1st Qu.:0.3285   1st Qu.:-1.602  
 Median :0.6250   Median :-0.9381   Median :0.3325   Median :-1.590  
 Mean   :0.6250   Mean   :-0.9380   Mean   :0.3321   Mean   :-1.589  
 3rd Qu.:0.6875   3rd Qu.:-0.9325   3rd Qu.:0.3359   3rd Qu.:-1.576  
 Max.   :0.7500   Max.   :-0.9266   Max.   :0.3388   Max.   :-1.561  
     ci.ub             pvalue             sigma2      
 Min.   :-0.2925   Min.   :0.004182   Min.   :0.1518  
 1st Qu.:-0.2886   1st Qu.:0.004536   1st Qu.:0.1539  
 Median :-0.2864   Median :0.004786   Median :0.1548  
 Mean   :-0.2872   Mean   :0.004733   Mean   :0.1544  
 3rd Qu.:-0.2853   3rd Qu.:0.004964   3rd Qu.:0.1551  
 Max.   :-0.2851   Max.   :0.005087   Max.   :0.1552  

Code

#few details/not so good
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$fidelity_low <- ifelse(dataN14_control$treatment_fidelity == 1, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = fidelity_low)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.2390981	0.7235207	-1.178976	1.657173	0.7410488	0.3948987
0.51	0.2393356	0.7222680	-1.176284	1.654955	0.7403674	0.3946738
0.52	0.2395855	0.7210059	-1.173560	1.652731	0.7396677	0.3944398
0.53	0.2398484	0.7197338	-1.170804	1.650501	0.7389486	0.3941963
0.54	0.2401253	0.7184508	-1.168012	1.648263	0.7382087	0.3939422
0.55	0.2404170	0.7171571	-1.165185	1.646019	0.7374469	0.3936777
0.56	0.2407246	0.7158518	-1.162319	1.643768	0.7366619	0.3934018
0.57	0.2410490	0.7145341	-1.159412	1.641510	0.7358521	0.3931135
0.58	0.2413916	0.7132032	-1.156461	1.639244	0.7350157	0.3928121
0.59	0.2417535	0.7118584	-1.153463	1.636970	0.7341510	0.3924968
0.60	0.2421363	0.7104988	-1.150416	1.634688	0.7332560	0.3921663
0.61	0.2425415	0.7091238	-1.147316	1.632399	0.7323285	0.3918203
0.62	0.2429708	0.7077315	-1.144158	1.630099	0.7313658	0.3914564
0.63	0.2434261	0.7063212	-1.140938	1.627790	0.7303655	0.3910739
0.64	0.2439097	0.7048915	-1.137652	1.625472	0.7293243	0.3906712
0.65	0.2444239	0.7034410	-1.134295	1.623143	0.7282390	0.3902469
0.66	0.2449715	0.7019681	-1.130861	1.620804	0.7271058	0.3897990
0.67	0.2455555	0.7004711	-1.127343	1.618454	0.7259203	0.3893256
0.68	0.2461793	0.6989480	-1.123733	1.616092	0.7246777	0.3888246
0.69	0.2468469	0.6973965	-1.120025	1.613719	0.7233727	0.3882933
0.70	0.2475628	0.6958141	-1.116208	1.611333	0.7219988	0.3877291
0.71	0.2483322	0.6941981	-1.112271	1.608936	0.7205490	0.3871288
0.72	0.2491610	0.6925453	-1.108203	1.606525	0.7190148	0.3864888
0.73	0.2500564	0.6908521	-1.103989	1.604102	0.7173867	0.3858052
0.74	0.2510264	0.6891142	-1.099613	1.601665	0.7156534	0.3850734
0.75	0.2520810	0.6873271	-1.095055	1.599217	0.7138014	0.3842880

Code

summary(sensitivity_moderator) #mean = .244, p = .730

      ri_t             beta              se             ci.lb       
 Min.   :0.5000   Min.   :0.2391   Min.   :0.6873   Min.   :-1.179  
 1st Qu.:0.5625   1st Qu.:0.2408   1st Qu.:0.6978   1st Qu.:-1.162  
 Median :0.6250   Median :0.2432   Median :0.7070   Median :-1.143  
 Mean   :0.6250   Mean   :0.2440   Mean   :0.7065   Mean   :-1.141  
 3rd Qu.:0.6875   3rd Qu.:0.2467   3rd Qu.:0.7155   3rd Qu.:-1.121  
 Max.   :0.7500   Max.   :0.2521   Max.   :0.7235   Max.   :-1.095  
     ci.ub           pvalue           sigma2      
 Min.   :1.599   Min.   :0.7138   Min.   :0.3843  
 1st Qu.:1.614   1st Qu.:0.7237   1st Qu.:0.3884  
 Median :1.629   Median :0.7309   Median :0.3913  
 Mean   :1.629   Mean   :0.7297   Mean   :0.3907  
 3rd Qu.:1.643   3rd Qu.:0.7365   3rd Qu.:0.3933  
 Max.   :1.657   Max.   :0.7410   Max.   :0.3949  

Code

#lot of information
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$fidelity_high <- ifelse(dataN14_control$treatment_fidelity == 2, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = fidelity_high)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.7749085	0.3696022	0.0505016	1.499316	0.0360289	0.2134398
0.51	0.7735492	0.3693568	0.0496233	1.497475	0.0362320	0.2138227
0.52	0.7721827	0.3691073	0.0487456	1.495620	0.0364361	0.2142024
0.53	0.7708085	0.3688537	0.0478685	1.493749	0.0366411	0.2145788
0.54	0.7694262	0.3685957	0.0469919	1.491861	0.0368472	0.2149515
0.55	0.7680352	0.3683330	0.0461157	1.489955	0.0370543	0.2153202
0.56	0.7666349	0.3680655	0.0452398	1.488030	0.0372625	0.2156846
0.57	0.7652246	0.3677928	0.0443640	1.486085	0.0374718	0.2160444
0.58	0.7638036	0.3675146	0.0434882	1.484119	0.0376823	0.2163992
0.59	0.7623711	0.3672307	0.0426120	1.482130	0.0378939	0.2167486
0.60	0.7609261	0.3669408	0.0417354	1.480117	0.0381069	0.2170922
0.61	0.7594677	0.3666444	0.0408579	1.478077	0.0383213	0.2174295
0.62	0.7579947	0.3663412	0.0399792	1.476010	0.0385372	0.2177601
0.63	0.7565058	0.3660307	0.0390988	1.473913	0.0387547	0.2180833
0.64	0.7549997	0.3657125	0.0382163	1.471783	0.0389739	0.2183986
0.65	0.7534748	0.3653862	0.0373311	1.469619	0.0391951	0.2187054
0.66	0.7519293	0.3650511	0.0364422	1.467416	0.0394185	0.2190030
0.67	0.7503611	0.3647067	0.0355490	1.465173	0.0396442	0.2192906
0.68	0.7487679	0.3643524	0.0346503	1.462886	0.0398727	0.2195674
0.69	0.7471472	0.3639875	0.0337448	1.460550	0.0401043	0.2198326
0.70	0.7454959	0.3636112	0.0328311	1.458161	0.0403395	0.2200852
0.71	0.7438106	0.3632227	0.0319072	1.455714	0.0405788	0.2203241
0.72	0.7420874	0.3628212	0.0309709	1.453204	0.0408228	0.2205483
0.73	0.7403215	0.3624057	0.0300194	1.450624	0.0410725	0.2207566
0.74	0.7385076	0.3619751	0.0290494	1.447966	0.0413288	0.2209477
0.75	0.7366395	0.3615284	0.0280568	1.445222	0.0415930	0.2211201

Code

summary(sensitivity_moderator)

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.7366   Min.   :0.3615   Min.   :0.02806  
 1st Qu.:0.5625   1st Qu.:0.7476   1st Qu.:0.3641   1st Qu.:0.03397  
 Median :0.6250   Median :0.7573   Median :0.3662   Median :0.03954  
 Mean   :0.6250   Mean   :0.7567   Mean   :0.3660   Mean   :0.03946  
 3rd Qu.:0.6875   3rd Qu.:0.7663   3rd Qu.:0.3680   3rd Qu.:0.04502  
 Max.   :0.7500   Max.   :0.7749   Max.   :0.3696   Max.   :0.05050  
     ci.ub           pvalue            sigma2      
 Min.   :1.445   Min.   :0.03603   Min.   :0.2134  
 1st Qu.:1.461   1st Qu.:0.03731   1st Qu.:0.2158  
 Median :1.475   Median :0.03865   Median :0.2179  
 Mean   :1.474   Mean   :0.03870   Mean   :0.2177  
 3rd Qu.:1.488   3rd Qu.:0.04005   3rd Qu.:0.2198  
 Max.   :1.499   Max.   :0.04159   Max.   :0.2211  

Writing tasks

Code

## WRITING TASKS ###########
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric(),
  sigma2 = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$test <- as.factor(dataN14_control$writing_tests)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = test)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            sigma2 = RMA$sigma2[1],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue	sigma2
0.50	0.4653052	0.2605106	-0.0452863	0.9758966	0.0740786	0.2401265
0.51	0.4648472	0.2604370	-0.0455999	0.9752943	0.0742818	0.2402942
0.52	0.4643843	0.2603603	-0.0459125	0.9746811	0.0744851	0.2404571
0.53	0.4639162	0.2602803	-0.0462238	0.9740562	0.0746887	0.2406150
0.54	0.4634426	0.2601969	-0.0465340	0.9734191	0.0748924	0.2407674
0.55	0.4629630	0.2601099	-0.0468430	0.9727691	0.0750965	0.2409141
0.56	0.4624772	0.2600191	-0.0471509	0.9721053	0.0753008	0.2410547
0.57	0.4619847	0.2599243	-0.0474577	0.9714270	0.0755055	0.2411888
0.58	0.4614850	0.2598253	-0.0477633	0.9707332	0.0757106	0.2413160
0.59	0.4609775	0.2597218	-0.0480679	0.9700230	0.0759162	0.2414358
0.60	0.4604618	0.2596136	-0.0483716	0.9692952	0.0761223	0.2415478
0.61	0.4599371	0.2595004	-0.0486744	0.9685486	0.0763291	0.2416514
0.62	0.4594027	0.2593819	-0.0489764	0.9677819	0.0765367	0.2417460
0.63	0.4588579	0.2592577	-0.0492779	0.9669937	0.0767453	0.2418310
0.64	0.4583017	0.2591273	-0.0495785	0.9661819	0.0769548	0.2419054
0.65	0.4577331	0.2589907	-0.0498794	0.9653456	0.0771659	0.2419691
0.66	0.4571510	0.2588473	-0.0501804	0.9644825	0.0773786	0.2420211
0.67	0.4565542	0.2586966	-0.0504819	0.9635902	0.0775932	0.2420604
0.68	0.4559410	0.2585381	-0.0507843	0.9626664	0.0778102	0.2420860
0.69	0.4553101	0.2583712	-0.0510881	0.9617082	0.0780300	0.2420969
0.70	0.4546593	0.2581953	-0.0513941	0.9607128	0.0782532	0.2420920
0.71	0.4539867	0.2580097	-0.0517029	0.9596764	0.0784803	0.2420699
0.72	0.4532898	0.2578136	-0.0520156	0.9585953	0.0787122	0.2420292
0.73	0.4525658	0.2576063	-0.0523332	0.9574649	0.0789498	0.2419683
0.74	0.4518115	0.2573867	-0.0526572	0.9562801	0.0791943	0.2418855
0.75	0.4510230	0.2571538	-0.0529893	0.9550352	0.0794471	0.2417788

Code

summary(sensitivity_moderator)

      ri_t             beta              se             ci.lb         
 Min.   :0.5000   Min.   :0.4510   Min.   :0.2572   Min.   :-0.05299  
 1st Qu.:0.5625   1st Qu.:0.4555   1st Qu.:0.2584   1st Qu.:-0.05101  
 Median :0.6250   Median :0.4591   Median :0.2593   Median :-0.04913  
 Mean   :0.6250   Mean   :0.4588   Mean   :0.2591   Mean   :-0.04912  
 3rd Qu.:0.6875   3rd Qu.:0.4624   3rd Qu.:0.2600   3rd Qu.:-0.04723  
 Max.   :0.7500   Max.   :0.4653   Max.   :0.2605   Max.   :-0.04529  
     ci.ub            pvalue            sigma2      
 Min.   :0.9550   Min.   :0.07408   Min.   :0.2401  
 1st Qu.:0.9619   1st Qu.:0.07535   1st Qu.:0.2411  
 Median :0.9674   Median :0.07664   Median :0.2418  
 Mean   :0.9667   Mean   :0.07668   Mean   :0.2415  
 3rd Qu.:0.9719   3rd Qu.:0.07798   3rd Qu.:0.2420  
 Max.   :0.9759   Max.   :0.07945   Max.   :0.2421  

Code

#self-conducted
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$tests_self <- ifelse(dataN14_control$writing_tests == 0, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = tests_self)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue
0.50	-0.7002043	0.4530110	-1.588089	0.1876810	0.1221849
0.51	-0.6990671	0.4526691	-1.586282	0.1881482	0.1225103
0.52	-0.6979268	0.4523201	-1.584458	0.1886042	0.1228324
0.53	-0.6967834	0.4519634	-1.582615	0.1890487	0.1231512
0.54	-0.6956366	0.4515989	-1.580754	0.1894809	0.1234663
0.55	-0.6944863	0.4512259	-1.578873	0.1899003	0.1237775
0.56	-0.6933322	0.4508442	-1.576971	0.1903062	0.1240845
0.57	-0.6921741	0.4504532	-1.575046	0.1906980	0.1243871
0.58	-0.6910117	0.4500524	-1.573098	0.1910748	0.1246849
0.59	-0.6898447	0.4496412	-1.571125	0.1914359	0.1249777
0.60	-0.6886728	0.4492191	-1.569126	0.1917805	0.1252649
0.61	-0.6874955	0.4487853	-1.567098	0.1921075	0.1255464
0.62	-0.6863125	0.4483391	-1.565041	0.1924161	0.1258216
0.63	-0.6851232	0.4478798	-1.562951	0.1927050	0.1260902
0.64	-0.6839271	0.4474063	-1.560827	0.1929732	0.1263516
0.65	-0.6827236	0.4469178	-1.558666	0.1932193	0.1266053
0.66	-0.6815119	0.4464133	-1.556466	0.1934420	0.1268508
0.67	-0.6802914	0.4458914	-1.554222	0.1936397	0.1270874
0.68	-0.6790610	0.4453509	-1.551933	0.1938107	0.1273146
0.69	-0.6778198	0.4447903	-1.549593	0.1939532	0.1275315
0.70	-0.6765667	0.4442080	-1.547198	0.1940650	0.1277374
0.71	-0.6753004	0.4436022	-1.544745	0.1941439	0.1279312
0.72	-0.6740193	0.4429707	-1.542226	0.1941873	0.1281121
0.73	-0.6727219	0.4423112	-1.539636	0.1941922	0.1282788
0.74	-0.6714063	0.4416211	-1.536968	0.1941552	0.1284301
0.75	-0.6700704	0.4408973	-1.534213	0.1940724	0.1285643

Code

summary(sensitivity_moderator)

      ri_t             beta               se             ci.lb       
 Min.   :0.5000   Min.   :-0.7002   Min.   :0.4409   Min.   :-1.588  
 1st Qu.:0.5625   1st Qu.:-0.6930   1st Qu.:0.4449   1st Qu.:-1.576  
 Median :0.6250   Median :-0.6857   Median :0.4481   Median :-1.564  
 Mean   :0.6250   Mean   :-0.6855   Mean   :0.4477   Mean   :-1.563  
 3rd Qu.:0.6875   3rd Qu.:-0.6781   3rd Qu.:0.4507   3rd Qu.:-1.550  
 Max.   :0.7500   Max.   :-0.6701   Max.   :0.4530   Max.   :-1.534  
     ci.ub            pvalue      
 Min.   :0.1877   Min.   :0.1222  
 1st Qu.:0.1904   1st Qu.:0.1242  
 Median :0.1926   Median :0.1260  
 Mean   :0.1920   Mean   :0.1258  
 3rd Qu.:0.1939   3rd Qu.:0.1275  
 Max.   :0.1942   Max.   :0.1286  

Code

#standardized tests
sensitivity_moderator <- data.frame(
  ri_t = as.numeric(),
  beta = as.numeric(),
  se = as.numeric(),
  ci.lb = as.numeric(),
  ci.ub = as.numeric(),
  pvalue = as.numeric()
)

for (ri_t in seq(from = .50, to=.75, by=.01)) {
  ###compute effect sizes for writing quality
  # effect sizes from within-subjects data
  datT_es<- escalc(measure = "SMCR",
                   m1i=datT$m_post, m2i=datT$m_pre,
                   sd1i=datT$sd_pre, sd2i=datT$sd_post,
                   ni=datT$ni, ri=rep(ri_t, 14),
                   slab = datT$slab)
  
  datT_es$id <- datT$id
  
  datC_es <- escalc(measure="SMCR",
                    m1i=datC$m_post, m2i=datC$m_pre,
                    sd1i=datC$sd_pre, sd2i=datC$sd_post,
                    ni=datC$ni, ri=rep((ri_t+.14), 14),
                    slab = datC$slab)
  datC_es$id<-datC$id
  
  dat <- data.frame(yi = datT_es$yi - datC_es$yi, vi = datT_es$vi + datC_es$vi)
  
  dat$id <- c(1,1,1,2,2,3,3,4,4,5,6,6,7,8)
  dat$tests_norm <- ifelse(dataN14_control$writing_tests == 2, 1, 0)
  
  # compute the meta-analysis
  RMA <- rma.mv(yi, vi, data=dat, random = ~ 1 | id, mods = tests_norm)
  
  # save estimates for sensitivity analysis
  sensitivity_moderator <- sensitivity_moderator %>%
    add_row(ri_t = ri_t, 
            beta = RMA$beta[,1][2],
            se = RMA$se[2],
            ci.lb = RMA$ci.lb[2],
            ci.ub = RMA$ci.ub[2],
            pvalue = RMA$pval[2]
    )
}

sensitivity_moderator %>%
  kbl() %>%
  kable_styling(bootstrap_options = c("striped", "hover"))

ri_t	beta	se	ci.lb	ci.ub	pvalue
0.50	0.8935590	0.5604653	-0.2049329	1.992051	0.1108652
0.51	0.8926921	0.5604341	-0.2057386	1.991123	0.1111920
0.52	0.8918096	0.5603991	-0.2065525	1.990172	0.1115234
0.53	0.8909106	0.5603602	-0.2073752	1.989196	0.1118598
0.54	0.8899938	0.5603170	-0.2082074	1.988195	0.1122015
0.55	0.8890581	0.5602695	-0.2090500	1.987166	0.1125489
0.56	0.8881021	0.5602174	-0.2099039	1.986108	0.1129025
0.57	0.8871243	0.5601604	-0.2107699	1.985019	0.1132627
0.58	0.8861232	0.5600983	-0.2116493	1.983896	0.1136301
0.59	0.8850969	0.5600307	-0.2125431	1.982737	0.1140053
0.60	0.8840434	0.5599574	-0.2134530	1.981540	0.1143891
0.61	0.8829607	0.5598781	-0.2143803	1.980302	0.1147820
0.62	0.8818461	0.5597924	-0.2153269	1.979019	0.1151851
0.63	0.8806971	0.5597000	-0.2162947	1.977689	0.1155993
0.64	0.8795105	0.5596004	-0.2172861	1.976307	0.1160257
0.65	0.8782831	0.5594932	-0.2183035	1.974870	0.1164656
0.66	0.8770108	0.5593780	-0.2193500	1.973372	0.1169203
0.67	0.8756895	0.5592543	-0.2204288	1.971808	0.1173916
0.68	0.8743143	0.5591217	-0.2215440	1.970173	0.1178814
0.69	0.8728796	0.5589794	-0.2227000	1.968459	0.1183918
0.70	0.8713791	0.5588271	-0.2239019	1.966660	0.1189254
0.71	0.8698055	0.5586640	-0.2251559	1.964767	0.1194851
0.72	0.8681505	0.5584896	-0.2264691	1.962770	0.1200744
0.73	0.8664044	0.5583027	-0.2278488	1.960658	0.1206971
0.74	0.8645561	0.5581036	-0.2293069	1.958419	0.1213586
0.75	0.8625924	0.5578911	-0.2308541	1.956039	0.1220644

Code

summary(sensitivity_moderator)

      ri_t             beta              se             ci.lb        
 Min.   :0.5000   Min.   :0.8626   Min.   :0.5579   Min.   :-0.2309  
 1st Qu.:0.5625   1st Qu.:0.8732   1st Qu.:0.5590   1st Qu.:-0.2224  
 Median :0.6250   Median :0.8813   Median :0.5597   Median :-0.2158  
 Mean   :0.6250   Mean   :0.8802   Mean   :0.5595   Mean   :-0.2165  
 3rd Qu.:0.6875   3rd Qu.:0.8879   3rd Qu.:0.5602   3rd Qu.:-0.2101  
 Max.   :0.7500   Max.   :0.8936   Max.   :0.5605   Max.   :-0.2049  
     ci.ub           pvalue      
 Min.   :1.956   Min.   :0.1109  
 1st Qu.:1.969   1st Qu.:0.1130  
 Median :1.978   Median :0.1154  
 Mean   :1.977   Mean   :0.1158  
 3rd Qu.:1.986   3rd Qu.:0.1183  
 Max.   :1.992   Max.   :0.1221  

Computational Environment

R-Version and libraries used.

Code

sessionInfo()

R version 4.2.2 (2022-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22631)

Matrix products: default

locale:
[1] LC_COLLATE=German_Germany.utf8  LC_CTYPE=German_Germany.utf8   
[3] LC_MONETARY=German_Germany.utf8 LC_NUMERIC=C                   
[5] LC_TIME=German_Germany.utf8    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] skimr_2.1.5      xfun_0.36        kableExtra_1.3.4 metafor_3.8-1   
 [5] metadat_1.2-0    Matrix_1.5-1     haven_2.5.1      here_1.0.1      
 [9] lubridate_1.9.2  forcats_1.0.0    stringr_1.5.0    dplyr_1.1.0     
[13] purrr_1.0.1      readr_2.1.4      tidyr_1.3.0      tibble_3.1.8    
[17] ggplot2_3.4.1    tidyverse_2.0.0 

loaded via a namespace (and not attached):
 [1] tidyselect_1.2.0  repr_1.1.6        lattice_0.22-5    colorspace_2.0-3 
 [5] vctrs_0.5.2       generics_0.1.3    viridisLite_0.4.1 htmltools_0.5.4  
 [9] base64enc_0.1-3   yaml_2.3.6        utf8_1.2.2        rlang_1.1.1      
[13] pillar_1.8.1      glue_1.6.2        withr_2.5.0       lifecycle_1.0.3  
[17] munsell_0.5.0     gtable_0.3.1      rvest_1.0.3       htmlwidgets_1.6.1
[21] evaluate_0.19     knitr_1.41        tzdb_0.3.0        fastmap_1.1.0    
[25] fansi_1.0.3       highr_0.10        scales_1.2.1      webshot_0.5.4    
[29] jsonlite_1.8.4    mime_0.12         systemfonts_1.0.4 hms_1.1.2        
[33] digest_0.6.31     stringi_1.7.8     grid_4.2.2        rprojroot_2.0.3  
[37] mathjaxr_1.6-0    cli_3.6.1         tools_4.2.2       magrittr_2.0.3   
[41] pkgconfig_2.0.3   ellipsis_0.3.2    xml2_1.3.3        timechange_0.1.1 
[45] svglite_2.1.1     rmarkdown_2.19    httr_1.4.4        rstudioapi_0.14  
[49] R6_2.5.1          nlme_3.1-160      compiler_4.2.2